thread_ts,messages_json "1692360841.807399 ","[{""user"": ""aganivada"", ""timestamp"": ""1692360841.807399"", ""content"": ""<@U02SF36PVKL> is this a protected branch?""}, {""user"": ""mnirmal"", ""timestamp"": ""1692360934.316139"", ""content"": ""looks like its not <@U02BV2DGUKC> - I thought all hotfix branches are protectec by default""}, {""user"": ""mnirmal"", ""timestamp"": ""1692360935.988169"", ""content"": ""Thanks""}, {""user"": ""mnirmal"", ""timestamp"": ""1692360951.919069"", ""content"": ""Can you please mark it protected? hotfix/0.0.1.1\n""}, {""user"": ""aganivada"", ""timestamp"": ""1692360958.620069"", ""content"": ""Sure""}, {""user"": ""aganivada"", ""timestamp"": ""1692361223.907999"", ""content"": ""done <@U02SF36PVKL> ""}, {""user"": ""mnirmal"", ""timestamp"": ""1692368464.753329"", ""content"": ""<@U02BV2DGUKC> I got this error while trying to hotfix-finish on metrics manager -\n```[ERROR] Failed to execute goal com.amashchenko.maven.plugin:gitflow-maven-plugin:1.18.0:hotfix-finish (default-cli) on project metrics-manager-parent: hotfix-finish: remote: GitLab: You are not allowed to push code to protected branches on this project.\n[ERROR] To \n[ERROR] ! [remote rejected] main -> main (pre-receive hook declined)\n[ERROR] ! [remote rejected] 0.0.1.1 -> 0.0.1.1 (pre-receive hook declined)\n[ERROR] error: failed to push some refs to ''```""}, {""user"": ""aganivada"", ""timestamp"": ""1692374081.884879"", ""content"": ""checking <@U02SF36PVKL>""}, {""user"": ""aganivada"", ""timestamp"": ""1692374149.121809"", ""content"": ""can you try now? main branch of this project had a restriction on roles to merge to main I updated it to allow developers""}, {""user"": ""mnirmal"", ""timestamp"": ""1692374433.134509"", ""content"": ""Sure anil""}]" "1680068887.069559 ","[{""user"": ""ppant"", ""timestamp"": ""1680068887.069559"", ""content"": "" Has `terminal-core-stack` been removed from base-infra\u2019s core? Can\u2019t find its reference anywhere in but it is being imported in `cdk_app.py` so locally synth is failing for core stacks""}, {""user"": ""aganivada"", ""timestamp"": ""1680069017.225529"", ""content"": ""<@U0431DZTPJM> can you copy it from here locally to test ""}, {""user"": ""ppant"", ""timestamp"": ""1680069085.765189"", ""content"": ""Shall I commit this to develop branch along with changes for sqs lambda?""}, {""user"": ""aganivada"", ""timestamp"": ""1680070731.059979"", ""content"": ""<@U0431DZTPJM> this file should come automatically to develop when we close release/0.9.9 branch this weekend. if it can wait until then we could allow release process to bring this change in develop""}]" "1687245425.498079 ","[{""user"": ""pjha"", ""timestamp"": ""1687245425.498079"", ""content"": "" FYI We had removed inbound rule from the '', please connect to the VPN to login.""}, {""user"": ""aganivada"", ""timestamp"": ""1687245462.582199"", ""content"": ""<@U04JT69T00K> please post screenshot of the inbound rule that we removed""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687275641.831529"", ""content"": ""So, <@U04JT69T00K> - unless someone is on VPN they cannot ssh to INT bastion?""}, {""user"": ""aganivada"", ""timestamp"": ""1687276501.088689"", ""content"": ""Yes Rama""}, {""user"": ""aganivada"", ""timestamp"": ""1687276561.380789"", ""content"": ""Bastion host access is enabled only for p81 SG. So source ip needs to be from VPN""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687279430.046589"", ""content"": ""<@U02BV2DGUKC> - is the process of getting on to INT VPN described in a page anywhere? Should we integrate into onboarding pages?""}, {""user"": ""aganivada"", ""timestamp"": ""1687280108.310219"", ""content"": ""yes <@U026PMDB1ND>, this was added by <@U0431DZTPJM> ""}]" "1673386087.015369 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1673386087.015369"", ""content"": ""<@U040RCBPBEC> - are we planning to summarization of all the responses for a given open-text question in the next few releases or is that not a requirement?""}, {""user"": ""svummidi"", ""timestamp"": ""1673387030.630579"", ""content"": ""Under discussion <@U026PMDB1ND> - I updated <@U033PPLNFRU> about OpenAI - One of his major concern is data privacy. Below are the few action items, currently working:\n1. We want to try G2 comments summarization and test it for scale. Probably we can try our platform and release retro comments also. \n2. We want to confirm our ability to run the model in-house even if we don\u2019t use it initially.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673387749.822239"", ""content"": ""Sounds good. Thanks <@U040RCBPBEC>""}]" "1687152024.202269 ","[{""user"": ""aganivada"", ""timestamp"": ""1687152024.202269"", ""content"": ""<@U026PMDB1ND> I was debugging the dynamodb issue with stage deployment it looks like the id of ssm parameter got changed in this commit so now we have a mix of some stacks having old cdk key as \""`dynamodb-table` \"" and some with new key suffixed. Problem happens when we deploy any old stack that already has a SSM parameter stored. It fails with error stating it can't create another entry as key already exists. The cases where it is passing are most likely the new dynamodb tables which were always using the new format. How can we address this issue?""}, {""user"": ""aganivada"", ""timestamp"": ""1687173653.777269"", ""content"": ""tried the suggestion but it is delete-then-create which might not work for us""}, {""user"": ""aganivada"", ""timestamp"": ""1687175163.438799"", ""content"": ""We can try delete the ssm parameter from cdk -> deploy -> putback the ssm parameter with proper id and deploy again wherever stack deployment is failing. This might mean for sometime the ssm parameter may not be available.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687187397.416079"", ""content"": ""Hmm ok. <@U02BV2DGUKC> - does this mean the service stacks need to be redeployed as well?""}, {""user"": ""aganivada"", ""timestamp"": ""1687187920.963199"", ""content"": ""I don't think we'd have to redeploy service stack <@U026PMDB1ND> because we will be adding the same ssm parameter in same path in next deployment. We just have to make sure that during this activity no other service stack is deployed""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687188052.396359"", ""content"": ""Okay. Let's do it whenever it works.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687188228.206029"", ""content"": ""I don't remember what drove the change though :slightly_smiling_face:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687188268.581919"", ""content"": ""I mean whether it was needed or if it was a let's make it look nice type of change""}, {""user"": ""aganivada"", ""timestamp"": ""1687188270.751559"", ""content"": "":slightly_smiling_face: this was a good learning. something to watch out for during reviews""}]" "1680244373.213369 ","[{""user"": ""pjha"", ""timestamp"": ""1680244373.213369"", ""content"": ""<@U02BV2DGUKC> <@U026PMDB1ND> I was trying to deploy the tenancy with the route53 target as alb endpoint, but it's throwing error that 'service already exist'.\nAny idea ?""}, {""user"": ""pjha"", ""timestamp"": ""1680245716.772859"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1680246026.430049"", ""content"": ""Not sure <@U04JT69T00K> , do we have to destroy and deploy? Destroy and deploy could add to some downtime""}, {""user"": ""pjha"", ""timestamp"": ""1680246103.610399"", ""content"": ""yes, it will have downtime if we destroy .""}, {""user"": ""pjha"", ""timestamp"": ""1680246177.807879"", ""content"": ""I will once try modifying the service name""}, {""user"": ""aganivada"", ""timestamp"": ""1680246293.542989"", ""content"": ""Can we first try with experiments or if we want to use tenancy we can use another AWS account""}, {""user"": ""pjha"", ""timestamp"": ""1680246316.611279"", ""content"": ""i am trying with experiment itself""}, {""user"": ""aganivada"", ""timestamp"": ""1680500239.164769"", ""content"": ""<@U04JT69T00K> did we get this working?""}, {""user"": ""pjha"", ""timestamp"": ""1680500318.580039"", ""content"": ""No, I tried modifying the name, it did't work""}, {""user"": ""pjha"", ""timestamp"": ""1680693722.937989"", ""content"": ""<@U026PMDB1ND> <@U02BV2DGUKC> Issue is not fixed.""}, {""user"": ""pjha"", ""timestamp"": ""1680693802.796029"", ""content"": ""tried modifying the service name but that did't work""}, {""user"": ""aganivada"", ""timestamp"": ""1680694232.378959"", ""content"": ""<@U04JT69T00K> so there is no other option than to destroy and deploy service again?""}, {""user"": ""aganivada"", ""timestamp"": ""1680694248.716839"", ""content"": ""does it work if we deploy after destroying?""}, {""user"": ""pjha"", ""timestamp"": ""1680694385.756799"", ""content"": ""<@U02BV2DGUKC> I couldn't figure out any other solution than deploying after destroying.""}, {""user"": ""pjha"", ""timestamp"": ""1680694398.188999"", ""content"": ""deploying after destroying works""}, {""user"": ""aganivada"", ""timestamp"": ""1680694416.005759"", ""content"": ""ok""}]" "1684344456.651589 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1684344456.651589"", ""content"": "" - we need to make a few upgrades over the next couple of weeks (routine maintenance, not necessarily feature driven):\n\n1. Opensearch ES from 1.2 to 1.3 (and then to 2.x)\n a. int, stage and then prod\n2. Aurora serverless (depends on the migration from RDS to Aurora).\n a. When we create Aurora instance, please make them the same version as what is in prod (<@U02BV2DGUKC>, <@U03KLHDKL1H>)\n b. Then we'd migrate it to the latest version as a trial. After that we can attempt this in stage & prod without any risk.\nWe also need some tooling (shell scripts would do) for ES re-indexing after applying new mappings. I'll take care of this.""}, {""user"": ""aganivada"", ""timestamp"": ""1684382248.552169"", ""content"": ""sure thank you <@U026PMDB1ND> will add tasks for these.\n\n<@U03KLHDKL1H> /<@U04JT69T00K> can we check and confirm is the aurora version is same as that of prod?""}, {""user"": ""pjha"", ""timestamp"": ""1684386334.817989"", ""content"": ""<@U02BV2DGUKC> yes, aurora int is same as that of prod it's 13.8""}, {""user"": ""askumar"", ""timestamp"": ""1684386594.636269"", ""content"": ""Yeah for int and stage we kept same as Prod""}]" "1689226665.101559 ","[{""user"": ""aganivada"", ""timestamp"": ""1689226665.101559"", ""content"": ""<@U03DHUAJVMK> /<@U03NZ7Z52S2> there is a HF request on tenancy from <@U033PPLNFRU> - \n\nthis updates the default invitation expiry from 7 days to 30 days and has option to change it if required as a configuration. I made the change and tested in int. Let me know if I can deploy the build in stage for validation.""}, {""user"": ""araman"", ""timestamp"": ""1689229682.820249"", ""content"": ""Please go ahead and deploy <@U02BV2DGUKC>.Do you want us to run any tests for this change? cc <@U03NZ7Z52S2>""}, {""user"": ""aganivada"", ""timestamp"": ""1689229736.704599"", ""content"": ""Yes <@U03DHUAJVMK>, just a simple invitation flow to verify that the TTL is updated to 30 days. I will let you know after deployment.""}, {""user"": ""araman"", ""timestamp"": ""1689229748.850439"", ""content"": ""Sure, thanks""}, {""user"": ""bganganna"", ""timestamp"": ""1689230569.273489"", ""content"": ""<@U02BV2DGUKC> Do we need to test for all scenarios ?or customer inviting vendor flow is fine?""}, {""user"": ""aganivada"", ""timestamp"": ""1689231051.954419"", ""content"": ""Any invitation flow is fine <@U03NZ7Z52S2> we can start with collaborators invite flow because that is most frequent flow. Customer inviting vendor/vendor inviting customer will be good to have.\n\n<@U03DHUAJVMK> /<@U03NZ7Z52S2> build is deployed to stage ""}, {""user"": ""bganganna"", ""timestamp"": ""1689240295.194199"", ""content"": ""<@U02BV2DGUKC> Tested customer and vendor collaborator flow.""}, {""user"": ""bganganna"", ""timestamp"": ""1689240321.157509"", ""content"": ""& customer inviting the existing vendor""}, {""user"": ""aganivada"", ""timestamp"": ""1689240388.747409"", ""content"": ""Cool thanks a lot <@U03NZ7Z52S2> :raised_hands:""}, {""user"": ""bganganna"", ""timestamp"": ""1689240396.879169"", ""content"": ""but dint do onboarding , till invitation only""}, {""user"": ""aganivada"", ""timestamp"": ""1689240418.562379"", ""content"": ""Yeah this change is only when we send invites to users""}, {""user"": ""aganivada"", ""timestamp"": ""1689240436.082239"", ""content"": ""There is no change in onboarding flow""}, {""user"": ""bganganna"", ""timestamp"": ""1689240478.547119"", ""content"": ""Do we need to test whether the invite expires , in that case can we change TTL to 1 hr ?""}, {""user"": ""aganivada"", ""timestamp"": ""1689240810.822989"", ""content"": ""Not required, Invite expiry is controlled by auth0 so we can skip that. When user clicks on accept invitation to untill they reach out dashboard auth0 controls the flow. We just have to make sure expiration is set to a month now instead of 7 days""}, {""user"": ""aganivada"", ""timestamp"": ""1689247941.711659"", ""content"": ""<@U033PPLNFRU> HF is deployed in prod, now the invitation's default TTL should be 30 days instead of 7 days in prod.""}]" "1684818317.645079 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1684818317.645079"", ""content"": ""<@U026PMDB1ND> - actually i answered the pulse around 9.25 Am IST - As per screenshot it says i have responded but it did not show up \u2026it got rejected bcoz it was closed ?""}, {""user"": ""snangia"", ""timestamp"": ""1684842961.240939"", ""content"": ""AFAIR, the pulse was closed post 9:30am.\nanyone once pulse is closed, we can't answer as the button is hidden.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1684843377.122939"", ""content"": ""<@U03RSS0S76Y> - actually if u see the screenshot , it says I answered - actually i submitted the answers and it did not throw error that pulse is closed""}, {""user"": ""snangia"", ""timestamp"": ""1684848550.898149"", ""content"": ""will have to check why your answers were not recorded, the pulse was open around 9:25am""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684855975.666509"", ""content"": "" - can you please debug this and figure out what happened here?""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1684856152.439919"", ""content"": ""sure <@U026PMDB1ND>, let me check the DB""}, {""user"": ""mnirmal"", ""timestamp"": ""1684857950.454869"", ""content"": ""I think <@U0336QZAF98> its some sort of a race condition -\n```400 Bad Request from POST \n\texception.stacktrace\t\norg.springframework.web.reactive.function.client.WebClientResponseException$BadRequest: 400 Bad Request from POST \n\tat org.springframework.web.reactive.function.client.WebClientResponseException.create(WebClientResponseException.java:196)\n\tSuppressed: The stacktrace has been enhanced by Reactor, refer to additional information below: \nError has been observed at the following site(s):\n\t*__checkpoint \u21e2 400 from POST [DefaultWebClient]```\nPMS threw 400, and the time is 9:33 AM. I'm guessing PMS threw 400 because the pulse already expired.""}, {""user"": ""mnirmal"", ""timestamp"": ""1684858024.812769"", ""content"": ""The reason why you didn't see any error in UI is because we update the responses in PMS in a async manner, and the errors are only logged.""}, {""user"": ""mnirmal"", ""timestamp"": ""1684858130.655919"", ""content"": """"}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684858218.484719"", ""content"": ""Hmm.. looks like a gap to me in how we present this to the user.\n\nWe cannot have a situation where the user thinks they submitted the pulse but it is not recorded.""}, {""user"": ""mnirmal"", ""timestamp"": ""1684858366.098429"", ""content"": ""I think we can remove the check from PMS - to accept responses only if the pulse is active. We added this mostly to deal with sys tests. All the expiry logic for slack and teams can be handled in SC.\n\nI can create a ticket for this if it makes sense. <@U02D4DUKDQC> <@U03RSS0S76Y> <@U026PMDB1ND>""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1684859906.405919"", ""content"": ""<@U02SF36PVKL> transactionally we should mark the pulse instance as closed only after we updated all the user messages that the pulse instance is expired""}, {""user"": ""snangia"", ""timestamp"": ""1684860097.479029"", ""content"": ""<@U02D4DUKDQC> the event is published from pulse-manager for expiry to SC. Should SC return publish an event back?\nAlso, I remember we had a check on start pulse button which would check if pulse has expired whenever user would click on it.""}, {""user"": ""snangia"", ""timestamp"": ""1684860108.720719"", ""content"": ""cc: <@U02SF36PVKL>""}, {""user"": ""snangia"", ""timestamp"": ""1684860275.075689"", ""content"": ""``` try {\n SlackUserSurveyContext userSurveyContext =\n SlackUserSurveyContext.fromId(SlackActionType.getActionIdWithoutType(actionId));\n if (hasSurveyExpired(userSurveyContext)) {\n (\n SimpleMapMessage.withMessage(\""Survey expired for user\"")\n .withTenantId(userSurveyContext.getTenantId())\n .with(USER_EMAIL, userSurveyContext.getUserEmail()));\n updateMessage(userSurveyContext, slackBlockAction);\n }```""}, {""user"": ""snangia"", ""timestamp"": ""1684860811.701099"", ""content"": ""~<@U026PMDB1ND> can you please confirm on tenant id and pulse instance id~""}, {""user"": ""snangia"", ""timestamp"": ""1684861231.648649"", ""content"": ""I see this in logs:\n```Unable to process slack notification payload={\""type\"":\""block_actions\"",\""user\"":{\""id\"":\""U0336QZAF98\"",\""username\"":\""rsrinivasan\"",\""name\"":\""rsrinivasan\"",\""team_id\"":\""T02649E326S\""},\""api_app_id\"":\""A0336JA7A8Y\"",\""token\"":\""1XNHXpGfIhPsi7KKq6Ct2Qoa\"",\""container\"":{\""type\"":\""view\"",\""view_id\"":\""V059CG5939P\""},\""trigger_id\"":\""5305915176098.2208320104230.1408aca02d1e50e237300fdcd50fc785\"",\""team\"":{\""id\"":\""T02649E326S\"",\""domain\"":\""axiamatic\""},\""enterprise\"":null,\""is_enterprise_install\"":false,\""view\"":{\""id\"":\""V059CG5939P\"",\""team_id\"":\""T02649E326S\"",\""type\"":\""modal\"",\""blocks\"":[{\""type\"":\""section\"",\""block_id\"":\""HEADER_BLOCK\"",\""text\"":{\""type\"":\""plain_text\"",\""text\"":\"":wave: Hello seshan , Please answer a few questions regarding Retro Survey\"",\""emoji\"":true}},{\""type\"":\""divider\"",\""block_id\"":\""fkaY\""},{\""type\"":\""input\"",\""block_id\"":\""21280\"",\""label\"":{\""type\"":\""plain_text\"",\""text\"":\""Were the requirements clear at the time you picked up your tasks?\"",\""emoji\"":false},\""optional\"":false,\""dispatch_action\"":true,\""element\"":{\""type\"":\""radio_buttons\"",\""action_id\"":\""f6339917-79ec-4cab-8110-b557dbd65beb\"",\""options\"":[{\""text\"":{\""type\"":\""plain_text\"",\""text\"":\""N\\/A\"",\""emoji\"":false},\""value\"":\"":21280:2430:97201:U0336QZAF98:1001:1\""},{\""text\"":{\""type\"":\""plain_text\"",\""text\"":\""Hazy, needed to work with PM for clarity\"",\""emoji\"":false},\""value\"":\"":21280:2431:97201:U0336QZAF98:1001:1\""},{\""text\"":{\""type\"":\""plain_text\"",\""text\"":\""50-50\"",\""emoji\"":false},\""value\"":\"":21280:2432:97201:U0336QZAF98:1001:1\""},{\""text\"":{\""type\"":\""plain_text\"",\""text\"":\""Mostly clear\"",\""emoji\"":false},\""value\"":\"":21280:2433:97201:U0336QZAF98:1001:1\""},{\""text\"":{\""type\"":\""plain_text\"",\""text\"":\""Crystal clear\"",\""emoji\"":false},\""value\"":\"":21280:2434:97201:U0336QZAF98:1001:1\""}]}},{\""type\"":\""divider\"",\""block_id\"":\""74RVs\""},{\""type\"":\""input\"",\""block_id\"":\""21282\"",\""label\"":{\""type\"":\""plain_text\"",\""text\"":\""Please comment on requirement clarity, more so if the requirements were NOT mostly clear OR crystal clear.\"",\""emoji\"":false},\""optional\"":true,\""dispatch_action\"":false,\""element\"":{\""type\"":\""plain_text_input\"",\""action_id\"":\"":21282:-1:97201:U0336QZAF98:1001:1\"",\""initial_value\"":\""\"",\""multiline\"":true,\""max_length\"":2048,\""dispatch_action_config\"":{\""trigger_actions_on\"":[\""on_enter_pressed\""]}}},{\""type\"":\""divider\"",\""block_id\"":\""VMPh\""},{\""type\"":\""input\"",\""block_id\"":\""21284\"",\""label\"":{\""type\"":\""plain_text\"",\""text\"":\""What went well in this release?\"",\""emoji\"":false},\""optional\"":true,\""dispatch_action\"":false,\""element\"":{\""type\"":\""plain_text_input\"",\""action_id\"":\"":21284:-1:97201:U0336QZAF98:1001:1\"",\""initial_value\"":\""\"",\""multiline\"":true,\""max_length\"":2048,\""dispatch_action_config\"":{\""trigger_actions_on\"":[\""on_enter_pressed\""]}}},{\""type\"":\""divider\"",\""block_id\"":\""kFm\""}],\""private_metadata\"":\""{\\\""user_name\\\"":\\\""seshan\\\"",\\\""product_name\\\"":\\\""Retro Survey\\\"",\\\""button_action_id\\\"":\\\""0:97201:rsrinivasan@axiamatic.com:1001:14137:17203:1684972493807:1\\\"",\\\""channel_id\\\"":\\\""D043Q187QJ3\\\"",\\\""message_id\\\"":\\\""1684626896.625479\\\"",\\\""page_num\\\"":1,\\\""is_follow_up\\\"":false,\\\""user_survey_context\\\"":{\\\""pulse_survey_instance_id\\\"":97201,\\\""user_email\\\"":\\\""\\\"",\\\""user_persona\\\"":\\\""1001\\\"",\\\""pvt_id\\\"":14137,\\\""tenant_id\\\"":17203,\\\""expires_at_epoch_millis\\\"":1684972493807,\\\""version\\\"":1}}\"",\""callback_id\"":\""survey-efd23e67-df7d-4b2a-bcdc-e1e6f9b89c1a\"",\""state\"":{\""values\"":{\""21280\"":{\""f6339917-79ec-4cab-8110-b557dbd65beb\"":{\""type\"":\""radio_buttons\"",\""selected_option\"":{\""text\"":{\""type\"":\""plain_text\"",\""text\"":\""Mostly clear\"",\""emoji\"":false},\""value\"":\"":21280:2433:97201:U0336QZAF98:1001:1\""}}},\""21282\"":{\"":21282:-1:97201:U0336QZAF98:1001:1\"":{\""type\"":\""plain_text_input\"",\""value\"":null}},\""21284\"":{\"":21284:-1:97201:U0336QZAF98:1001:1\"":{\""type\"":\""plain_text_input\"",\""value\"":null}}}},\""hash\"":\""1684813820.Khx0gsFv\"",\""title\"":{\""type\"":\""plain_text\"",\""text\"":\""Product Value Survey\"",\""emoji\"":false},\""clear_on_close\"":false,\""notify_on_close\"":false,\""close\"":{\""type\"":\""plain_text\"",\""text\"":\""Cancel\"",\""emoji\"":false},\""submit\"":{\""type\"":\""plain_text\"",\""text\"":\""Next\"",\""emoji\"":false},\""previous_view_id\"":null,\""root_view_id\"":\""V059CG5939P\"",\""app_id\"":\""A0336JA7A8Y\"",\""external_id\"":\""\"",\""app_installed_team_id\"":\""T02649E326S\"",\""bot_id\"":\""B0350HVQSAC\""},\""actions\"":[{\""action_id\"":\""f6339917-79ec-4cab-8110-b557dbd65beb\"",\""block_id\"":\""21280\"",\""selected_option\"":{\""text\"":{\""type\"":\""plain_text\"",\""text\"":\""Mostly clear\"",\""emoji\"":false},\""value\"":\"":21280:2433:97201:U0336QZAF98:1001:1\""},\""type\"":\""radio_buttons\"",\""action_ts\"":\""1684813822.785407\""}]}```""}, {""user"": ""snangia"", ""timestamp"": ""1684861348.667569"", ""content"": """"}, {""user"": ""snangia"", ""timestamp"": ""1684861634.129479"", ""content"": ""it failed at parsing `f6339917` from action id: f6339917-79ec-4cab-8110-b557dbd65beb\n```public static SlackActionType fromActionId(String actionId) {\n List<String> actionIdParts = Splitter.on(SEPARATOR).splitToList(actionId);\n return fromTypeId(Integer.parseInt(actionIdParts.get(0)));\n }```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684861940.400299"", ""content"": ""hmm.. what is the source of the assumption that the first component of that id would be a number?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684863077.674129"", ""content"": """"}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684863228.808109"", ""content"": ""It is basically trying to break all of 9ec530b3-288b-4929-925a-a6206fc92b46 or f6339917-79ec-4cab-8110-b557dbd65beb by \"":\"" and parse as int.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684863390.922139"", ""content"": "" Looks like something has changed after 9.11 deployment.\n\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684863489.009209"", ""content"": ""Don't think this the main thing that affected Seshan's submission. Looks like we have a general problem.""}, {""user"": ""mnirmal"", ""timestamp"": ""1684864318.359299"", ""content"": ""<@U026PMDB1ND> <@U02D4DUKDQC> can you please deploy 0.0.18 SC on stage? I recently deployed 0.0.18.1, that might be causing this""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684864348.933029"", ""content"": ""This has been going on since 5/21 Mohith""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684864380.992779"", ""content"": ""Are you sure this is a new change (I am talking about the NumberFormatException)""}, {""user"": ""sjain"", ""timestamp"": ""1684865469.245539"", ""content"": ""<@U026PMDB1ND>, I believe this change was introduced recently . <@U02SF36PVKL> has already raised a hotfix for it. ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684865622.312609"", ""content"": ""Makes sense. <@U02SF36PVKL> wdyt?""}, {""user"": ""mnirmal"", ""timestamp"": ""1684870546.110469"", ""content"": ""yes <@U026PMDB1ND>, NumberFormatException is fixed. I got confused with someother exception I saw in logs.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684901736.767419"", ""content"": ""ok""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1684903015.910359"", ""content"": ""<@U02SF36PVKL> - just curious - this issue should happen for all users right ? Why it will happen only for my submission""}, {""user"": ""mnirmal"", ""timestamp"": ""1684903131.566779"", ""content"": ""<@U0336QZAF98> the issue shared by Sagarika has nothing to do with your submission issue, it's a different issue. And yes it will happen for all the users.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1684903256.264359"", ""content"": ""Makes sense Thank you <@U02SF36PVKL>""}, {""user"": ""snangia"", ""timestamp"": ""1684903416.223039"", ""content"": ""<@U02SF36PVKL> what was the issue pertaining to seshan's submission? race condition only?""}, {""user"": ""mnirmal"", ""timestamp"": ""1684903488.624839"", ""content"": ""Issue is pulse started before expiry and submission happened post expiry.""}, {""user"": ""snangia"", ""timestamp"": ""1684903591.216139"", ""content"": ""the logs you shared were of 9:33 submission, while <@U0336QZAF98> filled it around 9:21am according to logs, does it take that long for submission?""}, {""user"": ""mnirmal"", ""timestamp"": ""1684910206.550879"", ""content"": ""yep looks like that - . <@U0336QZAF98> can confirm""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1684914482.720559"", ""content"": ""i submitted around some where around 9.26 - Not sure what happened here""}]" "1689665941.234589 ","[{""user"": ""akasim"", ""timestamp"": ""1689665941.234589"", ""content"": ""Score provider deployment in INT has failed with following error. Is there any changes required in cdk?\n``` 1/17 |1:06:57 PM | CREATE_FAILED | AWS::ServiceDiscovery::Service | alb-score-provider-service-int/Service/CloudmapService (albscoreproviderserviceintServiceCloudmapService202C3F22) Service [srv-wg6dp3okaqnnye7q] already exists\nnew Service (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/jsii-kernel-ZDsraI/node_modules/@aws-cdk/aws-servicediscovery/lib/service.js:82:25)\n\\_ FargateService.enableCloudMap (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/jsii-kernel-ZDsraI/node_modules/@aws-cdk/aws-ecs/lib/base/base-service.js:434:33)\n\\_ new BaseService (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/jsii-kernel-ZDsraI/node_modules/@aws-cdk/aws-ecs/lib/base/base-service.js:179:18)\n\\_ new FargateService (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/jsii-kernel-ZDsraI/node_modules/@aws-cdk/aws-ecs/lib/fargate/fargate-service.js:34:9)\n\\_ new ApplicationLoadBalancedFargateService (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/jsii-kernel-ZDsraI/node_modules/@aws-cdk/aws-ecs-patterns/lib/fargate/application-load-balanced-fargate-service.js:60:24)\n\\_ Kernel._create (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/tmpkey06x0i/lib/program.js:9964:29)\n\\_ Kernel.create (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/tmpkey06x0i/lib/program.js:9693:29)\n\\_ KernelHost.processRequest (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/tmpkey06x0i/lib/program.js:11544:36)\n\\_ KernelHost.run (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/tmpkey06x0i/lib/program.js:11504:22)\n\\_ Immediate._onImmediate (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/tmpkey06x0i/lib/program.js:11505:46)\n\\_ process.processImmediate (node:internal/timers:476:21)\n 1/17 |1:06:57 PM | UPDATE_IN_PROGRESS | AWS::Logs::LogGroup | score-provider-log-group (scoreproviderloggroup56FD7ECD) Requested update requires the creation of a new physical resource; hence creating one.\n 2/17 |1:06:57 PM | UPDATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) \n 2/17 |1:06:57 PM | UPDATE_FAILED | AWS::Logs::LogGroup | score-provider-log-group (scoreproviderloggroup56FD7ECD) Resource creation cancelled\nnew LogGroup (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/jsii-kernel-ZDsraI/node_modules/@aws-cdk/aws-logs/lib/log-group.js:165:26)\n\\_ Kernel._create (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/tmpkey06x0i/lib/program.js:9964:29)\n\\_ Kernel.create (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/tmpkey06x0i/lib/program.js:9693:29)\n\\_ KernelHost.processRequest (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/tmpkey06x0i/lib/program.js:11544:36)\n\\_ KernelHost.run (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/tmpkey06x0i/lib/program.js:11504:22)\n\\_ Immediate._onImmediate (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/tmpkey06x0i/lib/program.js:11505:46)\n\\_ process.processImmediate (node:internal/timers:476:21)\n 2/17 |1:06:57 PM | UPDATE_FAILED | AWS::Logs::LogGroup | score-provider-aws-distro-log-group (scoreproviderawsdistrologgroup10A169F7) Resource creation cancelled\nnew LogGroup (/private/var/folders/z2/ck0dxxt92h57tl25syxjmtsc0000gn/T/jsii-kernel-ZDsraI/node_modules/@aws-cdk/aws-logs/lib/log-group.js:165:26)```\n\ncc: <@U02D4DUKDQC>""}, {""user"": ""askumar"", ""timestamp"": ""1689669292.991689"", ""content"": ""Does not look like a permission issue .\n<@U04JT69T00K> could you please help take a look ?""}, {""user"": ""aganivada"", ""timestamp"": ""1689669683.043929"", ""content"": ""<@U02HQ78V9A5> do we have the latest code? <@U04JT69T00K> I hope we merged the lb instead of ecs ip change for this service?""}, {""user"": ""pjha"", ""timestamp"": ""1689669812.037839"", ""content"": ""<@U02BV2DGUKC> code is merged to develop""}, {""user"": ""pjha"", ""timestamp"": ""1689669824.893059"", ""content"": ""<@U02HQ78V9A5> Please pull develop changes""}, {""user"": ""akasim"", ""timestamp"": ""1689670292.850609"", ""content"": ""I had pulled at around 11am today.""}, {""user"": ""akasim"", ""timestamp"": ""1689670348.179089"", ""content"": ""<@U04JT69T00K>\nCan you try to deploy?\n``` ./cdkh.sh deploy int devops ../score-provider-service 0.0.19-COLL-2690-SNAPSHOT```""}, {""user"": ""pjha"", ""timestamp"": ""1689671345.519619"", ""content"": ""<@U02HQ78V9A5> Can you look into this error\n\nException encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'redisCacheManager' defined in file [/opt/axm/com/axm/vms/scoring/provider/redis/RedisCacheManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisson' defined in class path resource [org/redisson/spring/starter/RedissonAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.redisson.api.Redisso""}, {""user"": ""pjha"", ""timestamp"": ""1689671383.742819"", ""content"": ""I am not getting *Service [srv-wg6dp3okaqnnye7q] already exists* error""}, {""user"": ""akasim"", ""timestamp"": ""1689671696.407009"", ""content"": ""sure <@U04JT69T00K>\nThanks""}, {""user"": ""askumar"", ""timestamp"": ""1689671721.785679"", ""content"": ""<@U02HQ78V9A5> seems like these configs \nare not getting injected correctly.""}, {""user"": ""askumar"", ""timestamp"": ""1689678855.353039"", ""content"": ""<@U02HQ78V9A5> did it work?""}, {""user"": ""akasim"", ""timestamp"": ""1689678917.691419"", ""content"": ""yes deployment went through, cdk changes were not present in Prashant's workspace hence it failed""}]" "1688577977.016289 ","[{""user"": ""hchintamreddy"", ""timestamp"": ""1688577977.016289"", ""content"": ""<@U0431DZTPJM> I\u2019m seeing the following exception with unit tests in pulse manager after updating axm-commons version\n\n```Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.\n22:49:10.062 [main] ERROR [{}] o.sp.bo.SpringApplication - Application run failed\norg.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssmMultiInstanceConfigClient': Invocation of init method failed; nested exception is java.lang.NullPointerException: region must not be null.\n\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160) ~[spring-beans-5.3.22.jar:5.3.22]\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:440) ~[spring-beans-5.3.22.jar:5.3.22]\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796) ~[spring-beans-5.3.22.jar:5.3.22]\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620) ~[spring-beans-5.3.22.jar:5.3.22]\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.jar:5.3.22]\n\tat org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar:5.3.22]\n\tat org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar:5.3.22]\n\tat org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar:5.3.22]\n\tat org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar:5.3.22]\n\tat org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955) ~[spring-beans-5.3.22.jar:5.3.22]\n\tat org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.22.jar:5.3.22]\n\tat org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.22.jar:5.3.22]\n\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147) ~[spring-boot-2.7.3.jar:2.7.3]\n\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734) ~[spring-boot-2.7.3.jar:2.7.3]\n\tat org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408) ~[spring-boot-2.7.3.jar:2.7.3]\n\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:308) ~[spring-boot-2.7.3.jar:2.7.3]\n\tat org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:132) ~[spring-boot-test-2.7.3.jar:2.7.3]\n\tat org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99) ~[spring-test-5.3.22.jar:5.3.22]\n\tat org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:124) ~[spring-test-5.3.22.jar:5.3.22]\n\tat org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:124) ~[spring-test-5.3.22.jar:5.3.22]\n\tat org.springframework.test.context.support.DependencyInjectionTestExecutionListener.injectDependencies(DependencyInjectionTestExecutionListener.java:118) ~[spring-test-5.3.22.jar:5.3.22]\n\tat org.springframework.test.context.support.DependencyInjectionTestExecutionListener.prepareTestInstance(DependencyInjectionTestExecutionListener.java:83) ~[spring-test-5.3.22.jar:5.3.22]\n\tat org.springframework.boot.test.autoconfigure.SpringBootDependencyInjectionTestExecutionListener.prepareTestInstance(SpringBootDependencyInjectionTestExecutionListener.java:43) ~[spring-boot-test-autoconfigure-2.7.3.jar:2.7.3]\n\tat org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:248) ~[spring-test-5.3.22.jar:5.3.22]\n\tat org.springframework.test.context.junit.jupiter.SpringExtension.postProcessTestInstance(SpringExtension.java:138) ~[spring-test-5.3.22.jar:5.3.22]\n\tat org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeTestInstancePostProcessors$8(ClassBasedTestDescriptor.java:363) ~[junit-jupiter-engine-5.8.2.jar:5.8.2]\n\tat org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.executeAndMaskThrowable(ClassBasedTestDescriptor.java:368) ~[junit-jupiter-engine-5.8.2.jar:5.8.2]\n\tat org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeTestInstancePostProcessors$9(ClassBasedTestDescriptor.java:363) ~[junit-jupiter-engine-5.8.2.jar:5.8.2]\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]\n\tat java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]\n\tat java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?]\n\tat java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]\n\tat java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]\n\tat java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312) ~[?:?]\n\tat java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735) ~[?:?]\n\tat java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) ~[?:?]\n\tat java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]\n...\n\tat org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) ~[junit-platform-engine-1.8.2.jar:1.8.2]\n\tat org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) ~[junit-platform-engine-1.8.2.jar:1.8.2]\n\tat org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.8.2.jar:1.8.2]\n\tat org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) ~[junit-platform-engine-1.8.2.jar:1.8.2]\n\tat org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) ~[junit-platform-engine-1.8.2.jar:1.8.2]\n\tat org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35) ~[junit-platform-engine-1.8.2.jar:1.8.2]\n\tat org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) ~[junit-platform-engine-1.8.2.jar:1.8.2]\n\tat org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54) ~[junit-platform-engine-1.8.2.jar:1.8.2]\n\tat org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220) ~[junit-platform-launcher-1.3.1.jar:1.3.1]\n\tat org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188) ~[junit-platform-launcher-1.3.1.jar:1.3.1]\n\tat org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202) ~[junit-platform-launcher-1.3.1.jar:1.3.1]\n\tat org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181) ~[junit-platform-launcher-1.3.1.jar:1.3.1]\n\tat org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128) ~[junit-platform-launcher-1.3.1.jar:1.3.1]\n\tat org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150) ~[surefire-junit-platform-2.22.2.jar:2.22.2]\n\tat org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124) ~[surefire-junit-platform-2.22.2.jar:2.22.2]\n\tat org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) ~[surefire-booter-2.22.2.jar:2.22.2]\n\tat org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) ~[surefire-booter-2.22.2.jar:2.22.2]\n\tat org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) ~[surefire-booter-2.22.2.jar:2.22.2]\n\tat org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) ~[surefire-booter-2.22.2.jar:2.22.2]\nCaused by: java.lang.NullPointerException: region must not be null.\n\tat software.amazon.awssdk.utils.Validate.paramNotBlank(Validate.java:156) ~[utils-2.16.78.jar:?]\n\tat software.amazon.awssdk.regions.Region.of(Region.java:136) ~[regions-2.16.78.jar:?]\n\tat software.amazon.awssdk.regions.Region.of(Region.java:132) ~[regions-2.16.78.jar:?]\n\tat com.axm.platform.commons.multiinstance.SsmMultiInstanceConfigClient.createSsmClient(SsmMultiInstanceConfigClient.java:46) ~[axm-commons-0.0.18-COLL-2698-SNAPSHOT.jar:0.0.18-COLL-2698-SNAPSHOT]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]\n\tat jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]\n\tat java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]\n\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) ~[spring-beans-5.3.22.jar:5.3.22]\n\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333) ~[spring-beans-5.3.22.jar:5.3.22]\n\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157) ~[spring-beans-5.3.22.jar:5.3.22]\n\t... 86 more```""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1688578207.517149"", ""content"": ""Quick couple of observations\n\n\u2022 A bean annotated function is returning void in com.axm.platform.commons.multiinstance.SsmMultiInstanceConfigClient\n``` @Bean\n public void createSsmClient() {```\n\u2022 Could we move the bean definition to @Configuration class instead of SsmMultiInstanceConfigClient which is a @Component\n\u2022 No aws credentials passed to SsmClient how will it work?""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1688579139.341549"", ""content"": ""```Subject: [PATCH] COLL-2698\n---\nIndex: src/main/java/com/axm/platform/commons/config/SsmMultiInstanceConfig.java\nIDEA additional info:\nSubsystem: com.intellij.openapi.diff.impl.patch.CharsetEP\n<+>UTF-8\n===================================================================\ndiff --git a/src/main/java/com/axm/platform/commons/config/SsmMultiInstanceConfig.java b/src/main/java/com/axm/platform/commons/config/SsmMultiInstanceConfig.java\nnew file mode 100644\n--- /dev/null\t(date 1688578929037)\n+++ b/src/main/java/com/axm/platform/commons/config/SsmMultiInstanceConfig.java\t(date 1688578929037)\n@@ -0,0 +1,26 @@\n+package com.axm.platform.commons.config;\n+\n+import java.util.Optional;\n+import org.springframework.context.annotation.Bean;\n+import org.springframework.context.annotation.Configuration;\n+import software.amazon.awssdk.regions.Region;\n+import software.amazon.awssdk.services.ssm.SsmClient;\n+\n+/** Configuration class defining beans for multi instance config. */\n+@Configuration\n+public class SsmMultiInstanceConfig {\n+ private static final String AWS_REGION = \""AWS_REGION\"";\n+ private static final String ENV_NAME = \""ENV_NAME\"";\n+\n+ @Bean(\""ssmClientMultiInstance\"")\n+ SsmClient ssmClientMultiInstance() {\n+ String region = Optional.ofNullable(System.getenv(AWS_REGION)).orElse(Region.US_WEST_2.id());\n+ return SsmClient.builder().region(Region.of(region)).build();\n+ }\n+\n+ @Bean\n+ EnvEnum envEnum() {\n+ return EnvEnum.fromStringValue(\n+ Optional.ofNullable(System.getenv(ENV_NAME)).orElse(EnvEnum.INT.toString()));\n+ }\n+}\nIndex: src/main/java/com/axm/platform/commons/multiinstance/SsmMultiInstanceConfigClient.java\nIDEA additional info:\nSubsystem: com.intellij.openapi.diff.impl.patch.CharsetEP\n<+>UTF-8\n===================================================================\ndiff --git a/src/main/java/com/axm/platform/commons/multiinstance/SsmMultiInstanceConfigClient.java b/src/main/java/com/axm/platform/commons/multiinstance/SsmMultiInstanceConfigClient.java\n--- a/src/main/java/com/axm/platform/commons/multiinstance/SsmMultiInstanceConfigClient.java\t(revision 479534facaae7434847c181d2bda661e1bf15dc1)\n+++ b/src/main/java/com/axm/platform/commons/multiinstance/SsmMultiInstanceConfigClient.java\t(date 1688578669011)\n@@ -3,14 +3,10 @@\n import com.axm.platform.commons.config.EnvEnum;\n import com.fasterxml.jackson.core.JsonProcessingException;\n import com.fasterxml.jackson.databind.ObjectMapper;\n-import javax.annotation.PostConstruct;\n import lombok.RequiredArgsConstructor;\n import lombok.extern.log4j.Log4j2;\n-import org.springframework.cache.CacheManager;\n import org.springframework.cache.annotation.Cacheable;\n-import org.springframework.context.annotation.Bean;\n import org.springframework.stereotype.Component;\n-import software.amazon.awssdk.regions.Region;\n import software.amazon.awssdk.services.ssm.SsmClient;\n import software.amazon.awssdk.services.ssm.model.GetParameterRequest;\n import software.amazon.awssdk.services.ssm.model.GetParameterResponse;\n@@ -24,32 +20,10 @@\n @RequiredArgsConstructor\n public class SsmMultiInstanceConfigClient implements MultiInstanceConfigClient<SsmMultiInstanceConfigClient> {\n \n- private static final String AWS_REGION = \""AWS_REGION\"";\n- private static final String ENV_NAME = \""ENV_NAME\"";\n private static final String PARAMETER_NAME_FORMAT = \""/%s/multi-instance-config/%s\"";\n private final AppYamlConfigReader appYamlConfigReader;\n-\n- private SsmClient ssmClient;\n- private EnvEnum environment;\n-\n- /**\n- * Function that runs once during application context initialization for creating SSM client.\n- */\n- @PostConstruct\n- @Bean\n- public void createSsmClient() {\n- String region = System.getenv(AWS_REGION);\n-\n- (\""Creating SSM client for region: {}\"", region);\n-\n- this.ssmClient = SsmClient.builder()\n- .region(Region.of(region))\n- .build();\n-\n- this.environment = EnvEnum.fromStringValue(System.getenv(ENV_NAME));\n-\n- (\""SSM client created successfully\"");\n- }\n+ private final SsmClient ssmClient;\n+ private final EnvEnum environment;\n \n @Override\n public boolean isEnabled() {```""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1688579434.926079"", ""content"": ""This patch seems to fix the problem , but we still need to take care of the AWS credentials""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1688584413.676919"", ""content"": ""SSM client should act like there's no config if it can't get hold of aws creds""}, {""user"": ""ppant"", ""timestamp"": ""1688615247.600839"", ""content"": ""<@U02D4DUKDQC> Can you push the branch with your changes to Gitlab? I can have a look. Also are the tests failing only on gitlab or locally also?""}, {""user"": ""ppant"", ""timestamp"": ""1688619804.693009"", ""content"": ""<@U02D4DUKDQC> Thanks for catching this, happened because AWS_REGION env var was not present. I should have added a check and passed the default us-west-2 if not present""}, {""user"": ""aganivada"", ""timestamp"": ""1688619947.144169"", ""content"": ""<@U0431DZTPJM> do we need to add env variable in application.yml?""}, {""user"": ""aganivada"", ""timestamp"": ""1688619976.875779"", ""content"": ""or if possible can we set default value?""}, {""user"": ""ppant"", ""timestamp"": ""1688619993.975379"", ""content"": ""No <@U02BV2DGUKC>. This comes from the ECS task definition""}, {""user"": ""ppant"", ""timestamp"": ""1688620006.346519"", ""content"": ""Hari\u2019s latest MR on plat-libs-commons added this check""}, {""user"": ""ppant"", ""timestamp"": ""1688620017.814389"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1688620163.989089"", ""content"": ""got it LGTM, by default it should never be null in ECS case. should we add another enum say DEFAULT or something like that instead <@U0431DZTPJM>?""}, {""user"": ""ppant"", ""timestamp"": ""1688620408.306259"", ""content"": ""We already have NON_EXISTING env enum in case ENV_NAME is not there in ECS. Will add this in MR comment""}]" "1679964500.610319 ","[{""user"": ""ppant"", ""timestamp"": ""1679964500.610319"", ""content"": ""<@U026PMDB1ND> <@U02D4DUKDQC> I had fixed this but forgot to merge the MR :grimacing:. Merged it now""}, {""user"": ""aganivada"", ""timestamp"": ""1679973981.274139"", ""content"": ""<@U0431DZTPJM> how often do we print this log? should we release a HF version for all the services using latest observability?""}, {""user"": ""ppant"", ""timestamp"": ""1679974015.740969"", ""content"": ""This will be printed only when the service is started/bounced""}, {""user"": ""aganivada"", ""timestamp"": ""1679974083.533539"", ""content"": ""ok then we should be good I guess, can we post any FYI message in engg channel so there is no confusion""}]" "1681234016.534809 ","[{""user"": ""aganivada"", ""timestamp"": ""1681234016.534809"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1681234094.164499"", ""content"": ""<@U0431DZTPJM> can we update the lambdas updated and deployed to integration env here. we can request individual teams for sanity?""}]" "1693280392.926099 ","[{""user"": ""aganivada"", ""timestamp"": ""1693280392.926099"", ""content"": ""<@U04JT69T00K> can we move ask-ai job to gitlab runner? We have 30% or less space available in shared runner, This was raised by Rama in one of the threads I lost track of it. Lets also check other costly pipelines that can be moved to GL-runner""}, {""user"": ""pjha"", ""timestamp"": ""1693285561.495969"", ""content"": ""<@U02BV2DGUKC> sure I will move it to the custom GL runner, also will check the other costly pipelines .""}]" "1677651752.301219 ","[{""user"": ""svummidi"", ""timestamp"": ""1677651752.301219"", ""content"": ""<@U02BV2DGUKC> - My email address already registered to one of the tenant (15328). I tried to send invite from a different tenant (17203). After getting invite trying to accept the invite it is guiding to register a new account but registration is failing without providing much explanation. Is it possible to guide the user to login instead of register if there is an existing account?\nIf we are trying to register new account for an already existing user, can we give more specific error message?\nPlease let me know if there are any existing tickets for these issues.""}, {""user"": ""aganivada"", ""timestamp"": ""1677668273.915649"", ""content"": ""<@U040RCBPBEC> when you try to invite from tenant 17203 is there a vendor org already added for this product or we need to register?\n\nwe have an issue today that same user cannot register twice and if user gets invitation to register on 2 completely different products then the registration will fail since we create users during registration. So far this hasn't been a major issue since we didnt have a case where same vendor user gets invited on different products. In some special cases we handled by sending invitation to or registering as a different user. cc: <@U03DHUAJVMK> <@U03NZ7Z52S2>""}, {""user"": ""bganganna"", ""timestamp"": ""1677670591.115479"", ""content"": ""<@U040RCBPBEC> Tracker for the same , <@U02BV2DGUKC> Ideally if the user exists, instead of taking to the register page, we should navigate to login page.""}, {""user"": ""aganivada"", ""timestamp"": ""1677671023.269729"", ""content"": ""<@U03NZ7Z52S2> this is for the case where an existing user is invited ona different product which has not been registered yet""}, {""user"": ""aganivada"", ""timestamp"": ""1677671090.468769"", ""content"": ""It was one of the onboarding flows but not prioritised earlier. cc: <@U02GC8SE18V>""}, {""user"": ""bganganna"", ""timestamp"": ""1677671210.523979"", ""content"": ""oh ok""}, {""user"": ""svummidi"", ""timestamp"": ""1677685458.972229"", ""content"": ""<@U02BV2DGUKC> - Both the tenants and the products already created. This is a common case for our internal customers if we want to access two tenants.\nAs part of accepting invite for the second tenant, instead of trying to register, if I just login, everything works very nicely. It shows list of tenants after login.\nIt will be good if we do all necessary checks and redirect the user appropriately.""}, {""user"": ""anair"", ""timestamp"": ""1677696647.877419"", ""content"": ""<@U02BV2DGUKC> if already registered user, should we not completely remove the option to Sign Up on the auth0 screen. Only keep the option to log-in?""}, {""user"": ""aganivada"", ""timestamp"": ""1677728711.628399"", ""content"": ""<@U033PPLNFRU> we have removed signup link in second page when users login (screen shot attached). If I am not wrong <@U040RCBPBEC> is mentioning about the accept invitation email for existing user where by default auth0 asks users to signup though the user already has an account and will return an ambiguous error instead user should click on \""already have an account login\"".\n\n\nThis seems like a known limitation in auth0 and they are suggesting workaround to just add user directly without sending invitation not sure if that would work for us ""}, {""user"": ""aganivada"", ""timestamp"": ""1677758362.405739"", ""content"": ""Basically if we do as auth0 is suggesting we don't send any invitation but rather just inform user that they have been added to an org. There is no accept/reject option for user. If this is ok we can make the change""}, {""user"": ""svummidi"", ""timestamp"": ""1677777757.071679"", ""content"": ""<@U02BV2DGUKC> <@U02Q02G09QX> If the Product Owner or Org Admin adding as an authorized user and user already familiar with Axiamatic, I don\u2019t think approval is necessary but it will be good to inform the user about the new grant.""}, {""user"": ""aganivada"", ""timestamp"": ""1677816980.145439"", ""content"": ""sure <@U040RCBPBEC>,\n\n<@U033PPLNFRU> can you confirm?""}]" "1688712712.693079 ","[{""user"": ""pjha"", ""timestamp"": ""1688712712.693079"", ""content"": ""Default is 6 hour, but we can change it, need to add '\""multi-instance-ttl-in-hours\""' in cdk.json""}, {""user"": ""aganivada"", ""timestamp"": ""1688712756.682009"", ""content"": ""is it every 6 hours? I deployed at about 10 am and instance got deleted at 12""}, {""user"": ""pjha"", ""timestamp"": ""1688712923.975999"", ""content"": ""currently we have schedule the cron every hour, let's say ttl is 1 hour and we have deployed at 10 it will be deleted at 11, but if we have deployed after 10 then it will be deleted at 12.""}, {""user"": ""pjha"", ""timestamp"": ""1688712951.537769"", ""content"": ""we can modify cron to check every 15 mnt or even less if required""}, {""user"": ""aganivada"", ""timestamp"": ""1688713319.502919"", ""content"": ""hmmm..... I think 1 hour itself is too less for default can we set it to 3 hours?""}, {""user"": ""aganivada"", ""timestamp"": ""1688713365.741939"", ""content"": ""especially when we have multiple services and deployment itself takes 15-20 minutes""}, {""user"": ""aganivada"", ""timestamp"": ""1688713399.936819"", ""content"": ""if cron job runs every hour then what is the 6 hour TTL?""}, {""user"": ""aganivada"", ""timestamp"": ""1688713732.182739"", ""content"": ""got it my bad had ttl set for 1 hour for tenancy :man-facepalming:""}]" "1680656767.358329 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1680656767.358329"", ""content"": ""<@U02BV2DGUKC> <@U0431DZTPJM> - we are not reloading log4j xml from cloud config, which is requiring us to bounce the service. Today I updated PM's log4j and waited for 8 minutes before bouncing. We need to fix this once and for all.""}, {""user"": ""aganivada"", ""timestamp"": ""1680664338.952249"", ""content"": ""ok <@U026PMDB1ND>, after our last exercise we saw that any service that has cloud bus enabled (full integration with cloud config) should load config, checked with collab team they mentioned all collab services have it enabled so we didn't individually check services :man-facepalming:. Will verify all collab services""}, {""user"": ""ppant"", ""timestamp"": ""1680667357.356929"", ""content"": ""Only the following collab services have cloud-config-bus right now - message broker, sentiment collector and slack integrator""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680667706.704379"", ""content"": ""<@U02BV2DGUKC> - can we work with someone on collab to get everything integrated with cloud config?""}, {""user"": ""aganivada"", ""timestamp"": ""1680668342.244039"", ""content"": ""sure Rama, will create jira tickets for other collab services""}]" "1678428860.996399 ","[{""user"": ""aganivada"", ""timestamp"": ""1678428860.996399"", ""content"": ""<@U02HCMTQU3W> /<@U02GC8SE18V>/ <@U04ARFN3WQP> / starting a separate thread on getorgs response:\n\nwe did some analysis on response of this call, the average response time of this API is ~300ms in int+stage . However, in prod case it is 5x times higher the main reason behind this is because of system_user, typically during get orgs call we fetch orgs of a user but this auth0 API is not returning the role of user in these orgs this is something we need in the response to show orgs that user can log in to. So, we are currently making one call for each org user is associated with to find the role of user and when user is associated with too many orgs the call exponentially slows down. If we remove the exception of system_user from the mix in prod today then the response of this API call will depend on number of orgs the user is associated with, each org will add upto 100ms to the response.\n\nThis is a known issue in auth0 solution/workaround that auth0 is proposing in thread will not work for us. if we want to speed up the API further we have couple of options:\n\n1. we can introduce caching of some sort in backend but I don't think it will have a significant impact since the call is made only once when user is logging in. Lets say we set a cache expiry of 5 mins the cache would typically expire every time (since user wont be logging in multiple times within a cache interval) and we may have to go fetch it every time. \n2. We store users/orgs and roles locally so we don't have to go to auth0 everytime, this may introduce challenge to keep the data in sync but response will be faster\nWe will also talk to auth0 to see if there are any other alternatives to fix the issue before we proceed with making changes on our side. Please let us know your thoughts. cc: <@U0431DZTPJM>""}, {""user"": ""rvaidya"", ""timestamp"": ""1678429502.638069"", ""content"": ""<@U02BV2DGUKC> Ideal would be to get the role info along with org info from auth0. But as you mentioned, if its not possible then I am just thinking about (expect system user like) how many users will have multiple org association ?\n\nCurrently, we have vendor users connected to 2/3 customers. Do we forsee that it will grow to lets say 10 or more customers per vendor in near(how near) future?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678429788.084069"", ""content"": ""<@U02BV2DGUKC> - can the metadata - role , orgid- be maintained on user metadata on permissions""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678429812.758539"", ""content"": ""Bcoz anyway we maintain permission for the user for that org""}, {""user"": ""aganivada"", ""timestamp"": ""1678429940.664409"", ""content"": ""> Currently, we have vendor users connected to 2/3 customers. Do we forsee that it will grow to lets say 10 or more customers per vendor in near(how near) future?\nagreed <@U02GC8SE18V> , in general this will only be an issue when vendors are associated with too many customers. in customers scenario this will never be a problem because it is highly unlikely that a customer collaborator is associated with multiple orgs. In case of vendor if we take system_user as a base if a vendor is associated with say 30+ customers then the response might take 3 secs. so this not an eminent problem immediately except for system_user""}, {""user"": ""aganivada"", ""timestamp"": ""1678430205.113009"", ""content"": ""> can the metadata - role , orgid- be maintained on user metadata on permissions\nyeah that is an option auth0 is proposing <@U0336QZAF98> and instead of permissions we could store that in app_metadata so we don't have to make our token very big. However, we have to check what is the easiest way to keep this data updated always. So checking with auth0 if they can fix the API instead or propose a flow to keep the data updated from auth0 side so we don't have to take that responsibility .""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678432814.279699"", ""content"": ""Agreed <@U02BV2DGUKC> as long as auth0 provides a proper solution - that will be best . But for immediate needs - we can think about this - plus instead of putting everything in user metadata to tokens , we can filter only to permissions so that it is same as today""}]" "1677768700.036509 ","[{""user"": ""rvaidya"", ""timestamp"": ""1677768700.036509"", ""content"": "" <@U02BV2DGUKC> While inviting a collaborator on the same domain mapped to the tenant i am getting :\n```\""error\"": {\n \""code\"": 749,\n \""message\"": \""Domain of some email(s) is not valid as per the publisher\"",\n \""errors\"": [\n {\n \""message\"": \""Domain of some email(s) is not valid as per the publisher\"",\n \""domain\"": \""Tenant\"",\n \""reason\"": null,\n \""identifier\"": null\n }\n ]\n }```\nThe issue is that tenancy is making this call to the core : HTTP GET \n\nSpecificTenantOnly should be *FALSE* and the tenantId to query publisher on domain could be 19190\nOR\nSpecificTenantOnly should be *TRUE* and the tenantId to query publisher on domain should be 0\n\nWe dont have tenant specific publishers.\n\nCan you fix this pls?""}, {""user"": ""aganivada"", ""timestamp"": ""1677768928.716099"", ""content"": ""Sure <@U02GC8SE18V>, \n\n<@U0431DZTPJM> can we look into this tenant id should be 0 sorry for not catching this in MR.""}, {""user"": ""ppant"", ""timestamp"": ""1677768952.835999"", ""content"": ""Sure <@U02BV2DGUKC> <@U02GC8SE18V>, making specific tenant = false""}, {""user"": ""aganivada"", ""timestamp"": ""1677768965.882879"", ""content"": ""<@U0431DZTPJM> for now to unblock let's deploy the stage version to int""}, {""user"": ""aganivada"", ""timestamp"": ""1677768984.066799"", ""content"": ""I mean the latest main version""}, {""user"": ""aganivada"", ""timestamp"": ""1677769001.544839"", ""content"": ""We can push this change to int after testing""}, {""user"": ""rvaidya"", ""timestamp"": ""1677770493.273219"", ""content"": ""<@U0431DZTPJM> did we trigger any deployment?""}, {""user"": ""ppant"", ""timestamp"": ""1677770516.929069"", ""content"": ""Merged the fix. Waiting for the pipeline to pass. Will trigger it then""}, {""user"": ""ppant"", ""timestamp"": ""1677771700.121119"", ""content"": ""<@U02GC8SE18V> deployed""}, {""user"": ""aganivada"", ""timestamp"": ""1677775313.654889"", ""content"": ""<@U0431DZTPJM> is this fixed after deployment?""}, {""user"": ""rvaidya"", ""timestamp"": ""1677810990.989379"", ""content"": ""Yes <@U02BV2DGUKC>""}]" "1685445029.544959 ","[{""user"": ""aganivada"", ""timestamp"": ""1685445029.544959"", ""content"": ""<@U028EDANJM9> <@U034RLJA97X> jira-task-container seems to be erroring out querying for non existing tenants is this expected? ""}, {""user"": ""sranjan"", ""timestamp"": ""1685447577.360959"", ""content"": ""<@U028EDANJM9> <@U02BV2DGUKC> We have entries for non-existing tenants in our webhook_config table , so whenever scheduledJob runs it does query tenancy. We need to delete those entries. Created a tracker for it - ""}, {""user"": ""aganivada"", ""timestamp"": ""1685449147.595029"", ""content"": ""<@U034RLJA97X> don't we have tenant coordination hooked to jira service?""}, {""user"": ""sranjan"", ""timestamp"": ""1685449616.770199"", ""content"": ""<@U02BV2DGUKC> tenant coordination is there . I think , It happened because while writing system test for Jira-service, initially was not deleting tenants created by system-test so tenant-coordination didnt get triggered .""}, {""user"": ""aganivada"", ""timestamp"": ""1685449934.618389"", ""content"": ""Ok got it, can we also check if there are similar logs in stage and prod as well?""}, {""user"": ""aganivada"", ""timestamp"": ""1685679664.477889"", ""content"": ""<@U034RLJA97X> disabled logging from io.awspring.cloud.messaging.listener in int to reduce excessive logging in int ""}]" "1681186785.522409 ","[{""user"": ""aganivada"", ""timestamp"": ""1681186785.522409"", ""content"": ""<@U0336QZAF98> can we turn on SSO as default for Haleon? we can do that on a call today cc: <@U033PPLNFRU>""}, {""user"": ""aganivada"", ""timestamp"": ""1681225255.460579"", ""content"": ""<@U0336QZAF98> enabled SSO invitation FF for haleon""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1681227129.652839"", ""content"": ""We need to enable this for lvsands - just a reminder in case i forget""}, {""user"": ""aganivada"", ""timestamp"": ""1681231373.368709"", ""content"": ""sure I added to the steps we are tracking""}, {""user"": ""aganivada"", ""timestamp"": ""1681231429.177639"", ""content"": """"}]" "1690790135.941269 ","[{""user"": ""askumar"", ""timestamp"": ""1690790135.941269"", ""content"": "" <@U02BV2DGUKC>\nPlease have a look at ILM policy setup, suggestion around setup and present states.\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690823736.964539"", ""content"": ""<@U03KLHDKL1H> - we should do some pricing analysis up front for the proposed solution. We could list a few different options.\n\nIn INT and Stage, we should have 15d of hot and rest should be cold.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690831606.504979"", ""content"": ""<@U028EDANJM9> - are we still writing product discovery events to ES?""}, {""user"": ""askumar"", ""timestamp"": ""1690858924.059409"", ""content"": ""sure <@U026PMDB1ND>""}, {""user"": ""gshenoy"", ""timestamp"": ""1690861239.859729"", ""content"": ""> are we still writing product discovery events to ES?\nyes <@U026PMDB1ND>, although its not enabled for any customers. We can revisit this. Cc <@U02HCMTQU3W>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690864827.178579"", ""content"": ""If we are not consuming this we could stop the flow and drop the index. We can always bring it back if a usecase shows up""}, {""user"": ""gshenoy"", ""timestamp"": ""1690867202.152569"", ""content"": ""sure <@U026PMDB1ND>, we can do that. We also want to deprecate ( remove ) aws batch processing and move it to event framework when required.""}, {""user"": ""askumar"", ""timestamp"": ""1690902143.548599"", ""content"": ""<@U026PMDB1ND> added projected cost with 2 configurations, please have a look.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690905768.280299"", ""content"": ""Will do""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690919476.895639"", ""content"": ""<@U03KLHDKL1H> - I think we should start with #2 and see if that's good enough. We are logging a lot of page view records which can go away if the PM team wants to incorporate a different analysis tool\n\nAlso, please note down the costs before the proposed changes for the sake of completeness.""}, {""user"": ""askumar"", ""timestamp"": ""1690951644.682729"", ""content"": ""Sure <@U026PMDB1ND> \nWould keep the cost records.""}]" "1694161461.533839 ","[{""user"": ""aganivada"", ""timestamp"": ""1694161461.533839"", ""content"": ""<@U03NZ7Z52S2> Deployed fixes related to user removal to int this should resolve\n\n\n\n\nPTAL and let me know if you still observe any issues I will mark these to qa-ready once we deploy to stage, root cause is same for all 3 but since use cases are different it is manifesting in different forms. cc: <@U02GC8SE18V>""}, {""user"": ""bganganna"", ""timestamp"": ""1694176113.873659"", ""content"": ""<@U02BV2DGUKC> i still see happening ,""}, {""user"": ""aganivada"", ""timestamp"": ""1694176180.920339"", ""content"": ""Ok looks like somewhere cache is not getting cleared, can you review the steps in ticket again? I thought I tried exact flow""}, {""user"": ""bganganna"", ""timestamp"": ""1694176298.654179"", ""content"": ""added one more step, we need to invite the new collaborator as PO for engagement""}, {""user"": ""bganganna"", ""timestamp"": ""1694176312.960939"", ""content"": ""please check""}, {""user"": ""aganivada"", ""timestamp"": ""1694176327.635289"", ""content"": ""Ok will try""}, {""user"": ""bganganna"", ""timestamp"": ""1694177008.297269"", ""content"": ""I tried to verify *PLAT-2542 ,* removed the user from admins page and added as PO for 3 products , but post onboarding i see only one product , i dont see auth is called eventhough inviteMerged is false <@U04ARFN3WQP> <@U02BV2DGUKC>""}, {""user"": ""bganganna"", ""timestamp"": ""1694177047.813219"", ""content"": ""64fb159fd99d9afeb6c1eb88e170b6ed""}, {""user"": ""aganivada"", ""timestamp"": ""1694177059.897319"", ""content"": ""<@U03NZ7Z52S2> is this an existing session?""}, {""user"": ""bganganna"", ""timestamp"": ""1694177114.950639"", ""content"": ""no i tried onboarding from diff browser""}, {""user"": ""aganivada"", ""timestamp"": ""1694177194.431469"", ""content"": ""Hmmm may be some ui wip build? It was working a while ago (Info call) <@U04ARFN3WQP> please check and let me know if there is any backend issue""}, {""user"": ""bganganna"", ""timestamp"": ""1694177222.019789"", ""content"": ""ok""}, {""user"": ""amishra"", ""timestamp"": ""1694178134.420619"", ""content"": ""<@U03NZ7Z52S2> quick call?""}, {""user"": ""amishra"", ""timestamp"": ""1694178506.311129"", ""content"": ""can you please share creds for that account so that i can reproduce it?""}, {""user"": ""aganivada"", ""timestamp"": ""1694404507.525589"", ""content"": ""<@U04ARFN3WQP> /<@U03NZ7Z52S2> any luck with this?\n\n<@U03NZ7Z52S2> checking on PLAT-2540""}, {""user"": ""aganivada"", ""timestamp"": ""1694409323.788489"", ""content"": ""<@U03NZ7Z52S2> not able to re-produce PLAT-2540 do you have few mins?""}, {""user"": ""bganganna"", ""timestamp"": ""1694410416.256739"", ""content"": ""Will test both issues in stage and update <@U02BV2DGUKC> <@U04ARFN3WQP>""}, {""user"": ""bganganna"", ""timestamp"": ""1694415468.878459"", ""content"": ""** worked fine in stage, I could see older association got deleted in invitation_permissions and invitation table, invite_merged was false initially <@U04ARFN3WQP> <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1694417214.998129"", ""content"": ""cool thank you <@U03NZ7Z52S2>, marking 2541 & 2542 as qa-ready. 2540 I will update after debugging""}]" "1675219088.480769 ","[{""user"": ""araman"", ""timestamp"": ""1675219088.480769"", ""content"": ""This is not collaborator assignment flow but inviting existing vendor""}, {""user"": ""aganivada"", ""timestamp"": ""1675219902.684439"", ""content"": ""<@U03DHUAJVMK> you mean onboarding based on invitation? did onboarding flow trigger?""}, {""user"": ""aganivada"", ""timestamp"": ""1675220319.043789"", ""content"": ""verified for another flow where permissions seem to be updated properly ""}, {""user"": ""araman"", ""timestamp"": ""1675224943.941189"", ""content"": ""We can discuss over a call when. You are available""}]" "1688367311.116169 ","[{""user"": ""nsrivastava"", ""timestamp"": ""1688367311.116169"", ""content"": "" <@U02BV2DGUKC> <@U04JT69T00K> do we have an axm repo of docker images, need an image for gl pipeline of teams-integrator (node js app) with node, maven and netcat installed. I can publish one, just want to know if there is a repo or there already exists a file that can be modified""}, {""user"": ""aganivada"", ""timestamp"": ""1688368842.545659"", ""content"": ""<@U03RQDE3QUS> we do publish custom docker images. here are some examples:\n\n\n\n\nyou can check pipeline of any of these jobs, once the image is published it will be in \n\nwould this help?""}, {""user"": ""aganivada"", ""timestamp"": ""1688368885.268349"", ""content"": ""if the changes/libs are minimal we can use base-docker-image otherwise we may need to create a new one""}, {""user"": ""nsrivastava"", ""timestamp"": ""1688368989.960459"", ""content"": ""thanks <@U02BV2DGUKC>, this helps. need to topup base image with node for now, I think we can update the base one because the same can be used later when any other service need to publish js client, what do you suggest""}, {""user"": ""aganivada"", ""timestamp"": ""1688369136.492249"", ""content"": ""since the base image is used for all services the image size might increase while we dont use much of it. Small os libraries or updates to existing libraries are fine but since node can add a lot of unnecessary packages which are not useful for other services I think it might be better to spin a new image.""}, {""user"": ""nsrivastava"", ""timestamp"": ""1688447533.650919"", ""content"": ""<@U02BV2DGUKC> <@U04JT69T00K> the js and java are building together now, facing 401 however while pulling axm parent , this is the branch \n\nand failed job log\n,\n\nPlease take a look if something is missing""}, {""user"": ""aganivada"", ""timestamp"": ""1688448098.349939"", ""content"": ""<@U026PMDB1ND> We are getting unresolvable host error while attempting to download axm-parent (verified axm-parent settings ideally parent should authorize projects with valid token.) is there any other setting to check for new projects? \n\n```1109 [INFO] Scanning for projects...\nDownloading from gitlab-maven: \nDownloading from central: \n1954 [ERROR] [ERROR] Some problems were encountered while processing the POMs:\n[FATAL] Non-resolvable parent POM for com.axm.collaboration:teams-integrator-specification:0.0.1: Could not transfer artifact com.axm.platform:axm-parent:pom:0.0.18 from/to gitlab-maven (): authentication failed for , status: 401 Unauthorized and 'parent.relativePath' points at no local POM @ line 9, column 11```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1688448777.439969"", ""content"": ""<@U03RQDE3QUS> - sometime back we had some issues with the slack integrator project w/ maven and pulling builds. Do you remember? I vaguely remember having to make some tweaks to get the build to pass""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1688449105.744719"", ""content"": ""<@U02BV2DGUKC> - which branch is broken? Are we able to build it locally (on your machine)?""}, {""user"": ""aganivada"", ""timestamp"": ""1688449152.253149"", ""content"": ""I haven't tried on my machine Rama, Nayan said builds work fine locally. <@U03RQDE3QUS> can you confirm?""}, {""user"": ""aganivada"", ""timestamp"": ""1688449188.741519"", ""content"": ""Branch -> ""}, {""user"": ""nsrivastava"", ""timestamp"": ""1688449250.199939"", ""content"": ""<@U026PMDB1ND>, the slack-integrator issue I think we had included the plugin repository in pom, I tried that and din\u2019t work also it was not 401 I think. This commit had that change ""}, {""user"": ""nsrivastava"", ""timestamp"": ""1688449265.810629"", ""content"": ""Yeah, it works locally <@U02BV2DGUKC> <@U026PMDB1ND>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1688449280.081519"", ""content"": ""I can't build feature/COLL-2683 locally""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1688449316.715839"", ""content"": ""Do we need to do more than mvn clean install, <@U03RQDE3QUS>?""}, {""user"": ""nsrivastava"", ""timestamp"": ""1688449373.109159"", ""content"": ""yes <@U026PMDB1ND>, if the error is spec file not found, we need to do\n1. npm install\n2. npm run start &\n3. mvn clean install""}, {""user"": ""nsrivastava"", ""timestamp"": ""1688449384.647449"", ""content"": ""npm run start builds the spec file""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1688449595.742169"", ""content"": ""Do we have the same steps in gitlab-ci.yml?""}, {""user"": ""nsrivastava"", ""timestamp"": ""1688449635.744819"", ""content"": ""yes <@U026PMDB1ND>, the pre-requisites are executing now, I verified the generated files in the pipeline""}, {""user"": ""nsrivastava"", ""timestamp"": ""1688451338.173259"", ""content"": ""thanks <@U026PMDB1ND>, the java client generate fine. The publish block is copied from UI, may need some more tweaking, will check.""}, {""user"": ""aganivada"", ""timestamp"": ""1688451528.116889"", ""content"": ""<@U03RQDE3QUS> / <@U026PMDB1ND> what was the change? Just for my understanding""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1688451831.425289"", ""content"": ""`mvn clean install` --> `mvn clean install -s $CI_SETTINGS` \n\n<@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1688451864.721669"", ""content"": ""ok thank you <@U026PMDB1ND>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1688451906.507039"", ""content"": ""We should get to this:\n\n`mvn $MAVEN_CLI_OPTS clean install -s $CI_SETTINGS`\n\nto be same as everything else.""}, {""user"": ""nsrivastava"", ""timestamp"": ""1688451935.483169"", ""content"": ""sure <@U026PMDB1ND>, will update.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1688452410.077599"", ""content"": ""cool""}]" "1689838654.208169 ","[{""user"": ""kagarwal"", ""timestamp"": ""1689838654.208169"", ""content"": ""Can someone please give me access to our Gitlab repo - \n\n\ncc ""}, {""user"": ""kagarwal"", ""timestamp"": ""1689839001.325729"", ""content"": ""cc <@U0281D3GWHL>""}, {""user"": ""aganivada"", ""timestamp"": ""1689839204.964339"", ""content"": ""<@U05DR9X8QT1> sent invitation to join axiamatic-main in gitlab""}, {""user"": ""kagarwal"", ""timestamp"": ""1689839297.118849"", ""content"": ""Thanks, got it.""}]" "1679289980.990969 ","[{""user"": ""ppant"", ""timestamp"": ""1679289980.990969"", ""content"": "" <@U028EDANJM9> <@U034RLJA97X> We were looking into jira lambda integrations with VPC and I came across which is used in jira lambda for auth and common interfaces. The methods this project contains are almost similar to . Any specific reason why we are using data-brokering-libs-jira here instead of platform-libs-epoch? Asking this because lambda VPC integration will become much easier if we start using a common library for shared interfaces/methods""}, {""user"": ""gshenoy"", ""timestamp"": ""1679291147.900829"", ""content"": ""No specific reason <@U0431DZTPJM>. Since the epoch library is specific to epoch, we didn\u2019t add it as a dependency. We can refactor the common methods into a plat-commons like lib if required.\nCc <@U034RLJA97X>""}, {""user"": ""ppant"", ""timestamp"": ""1679291399.245659"", ""content"": ""Thanks <@U028EDANJM9>. <@U02BV2DGUKC> I think if we can rename plat-libs-common to plat-lambda-commons (or create a new one and deprecate this one eventually) things will be more streamlined. The problem can come if in one lambda there is a typo in env var for endpoints (like `TENANCY_SERVICE_URL` instead of `TENANCY_URL`) as these env vars are named using the naming scheme in base-infra""}, {""user"": ""sranjan"", ""timestamp"": ""1679293282.803679"", ""content"": ""<@U028EDANJM9> should we create tracker ticket for this?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679330979.055829"", ""content"": ""I think we could go either way (renaming or copying), but renaming might help us converge must faster, <@U0431DZTPJM>""}, {""user"": ""ppant"", ""timestamp"": ""1679359682.175079"", ""content"": ""Sure <@U026PMDB1ND>, we have decided to rename `plat-libs-epoch` to `plat-libs-lambda` and use it in all lambdas to make things streamlined""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679359759.268689"", ""content"": ""Maybe it is better to rename it plat-libs-py-common or something because we could use it in non-lambda cases as well.""}]" "1676641211.273499 ","[{""user"": ""nsrivastava"", ""timestamp"": ""1676641211.273499"", ""content"": "" the docker task in SC main branch failed with error\n```Will publish to ECR\n439An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\n\n441\nCleaning up project directory and file based variables\n00:01\n443ERROR: Job failed: exit code 1```\nhas anyone seen this before or could this be intermittent ?""}, {""user"": ""nsrivastava"", ""timestamp"": ""1676641616.863979"", ""content"": ""looks intermittent, retry helped""}, {""user"": ""aganivada"", ""timestamp"": ""1676645679.522159"", ""content"": ""Weird ....... wonder if gitlab again changed their certificate""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1676645717.869699"", ""content"": ""That is the most likely explanation""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1676645735.808789"", ""content"": ""Friday afternoon in EU to screw around a bit, I guess :slightly_smiling_face:""}]" "1673241490.412819 ","[{""user"": ""aganivada"", ""timestamp"": ""1673241490.412819"", ""content"": ""<@U0431DZTPJM> assigned to you currently marked this for next sprint as we have only 2 days left but if you have bandwidth please feel free to pick it up cc: <@U02T4E0BVA8>""}, {""user"": ""sparashar"", ""timestamp"": ""1673241684.342989"", ""content"": ""cc <@U02GC8SE18V>\nThe Apps service would be using this API""}]" "1682350368.511449 ","[{""user"": ""pjha"", ""timestamp"": ""1682350368.511449"", ""content"": ""<@U02BV2DGUKC> <@U026PMDB1ND> I was trying to deploy slack-integrator-service with route53 mapping to loadbalancer dns, but it's throughing an error""}, {""user"": ""pjha"", ""timestamp"": ""1682350495.004269"", ""content"": """"}, {""user"": ""pjha"", ""timestamp"": ""1682350550.093419"", ""content"": ""Here we can see the length InstanceId is more than 64""}, {""user"": ""pjha"", ""timestamp"": ""1682350639.518499"", ""content"": ""here we are using *aws_cdk.aws_servicediscovery.Service.register_load_balancer(_id_, _load_balancer_, _custom_attributes=None_)* \nwe can't provide instance_id as an arguments to the method""}, {""user"": ""aganivada"", ""timestamp"": ""1682351324.955899"", ""content"": ""<@U04JT69T00K> can we abbreviate instanceid to have someting like LB instead of loadbalancer? also \""slackintegrator\"" seems to be repeating is this expected?""}, {""user"": ""pjha"", ""timestamp"": ""1682354727.852819"", ""content"": ""<@U02BV2DGUKC> Deployed tenancy with the changes that you suggested, it succeeded without deleting the stack.""}, {""user"": ""aganivada"", ""timestamp"": ""1682355219.755619"", ""content"": ""cool thank you <@U04JT69T00K> can we submit a MR merging change to release/0.9.10""}, {""user"": ""pjha"", ""timestamp"": ""1682355379.965049"", ""content"": ""I have merged to develop branch since slack-integrator-service-changes was there, should I create merge request from develop to release/0.9.10""}, {""user"": ""pjha"", ""timestamp"": ""1682355380.489359"", ""content"": ""?""}, {""user"": ""aganivada"", ""timestamp"": ""1682355463.819669"", ""content"": ""for stage deployment we have to use \""release/0.9.10\"" only this week. Can we cherry pick changes to release/0.9.10?""}, {""user"": ""pjha"", ""timestamp"": ""1682355521.815129"", ""content"": ""sure""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682355534.604529"", ""content"": ""How did we pick the instance id eventually?""}, {""user"": ""aganivada"", ""timestamp"": ""1682355572.391179"", ""content"": ""<@U026PMDB1ND> we changes \""loadbalancer\"" in instanceid to lb""}, {""user"": ""aganivada"", ""timestamp"": ""1682355587.097559"", ""content"": ""Prashant can you share the new instanceid?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682355591.321259"", ""content"": ""is this done across the board?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682355597.586069"", ""content"": ""at a base level?""}, {""user"": ""pjha"", ""timestamp"": ""1682355641.837939"", ""content"": """"}, {""user"": ""pjha"", ""timestamp"": ""1682355651.648919"", ""content"": """"}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682355674.968139"", ""content"": ""Which file is the last screenshot from?""}, {""user"": ""pjha"", ""timestamp"": ""1682355692.193539"", ""content"": ""\""core/commons/base_app_stack.py\""""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682355730.770489"", ""content"": ""Where do we build the rest of the string?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682355780.560109"", ""content"": ""slackintegratorstackintslackintegratorsrevice doesn't seem to be very useful. Wonder if we can shorten it""}, {""user"": ""pjha"", ""timestamp"": ""1682355875.758019"", ""content"": ""not sure how it's constructing the InstanceId, I will look into it""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682355887.798619"", ""content"": ""ok""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682356072.667989"", ""content"": ""```>>> len(\""slackintegratorstackproductionslackintegratorservicelb3BA97937\"")\n62```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682356145.157219"", ""content"": ""```>>> len(\""sentimentcollectorstackproductionsentimentcollectorservicelb3BA97937\"")\n68```""}, {""user"": ""pjha"", ""timestamp"": ""1682359584.711999"", ""content"": ""InstanceId is formed using \""construct_id\"", & \""service-id\"" as depicted below. I think shorting service_id should would do.""}, {""user"": ""pjha"", ""timestamp"": ""1682359665.699609"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1682394820.456489"", ""content"": ""<@U04JT69T00K> if we are attempting to shorten the service_id can we track this as a separate change and submit MR to develop so we'd know if there are any side-effects of this change from CDK during deploy? cc: <@U026PMDB1ND>""}]" "1672978506.456839 ","[{""user"": ""slackbot"", ""timestamp"": ""1672978506.456839"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1672978669.261199"", ""content"": "" jfyi\n\n<@U0336QZAF98> will this have any impact on SSO accounts already setup in int and stage? I dont see auth0 calling out anything on SAML in the mail so I think integration should not break due to this change.""}]" "1690518421.919599 ","[{""user"": ""aganivada"", ""timestamp"": ""1690518421.919599"", ""content"": ""<@U03KLHDKL1H> can you look into auth0 is case sensitive while sending invitation but once user accepts invitation they lowercase the email. so to keep things consistent we can lowercase emails before sending it to auth0. cc: <@U02GC8SE18V> <@U03NZ7Z52S2>""}, {""user"": ""askumar"", ""timestamp"": ""1690518461.412029"", ""content"": ""Sure""}]" "1693403669.267489 ","[{""user"": ""aganivada"", ""timestamp"": ""1693403669.267489"", ""content"": ""<@U04JT69T00K> can we sync up on aws network diagram I experimented with earlier may be we can use it to generate the diagram""}, {""user"": ""pjha"", ""timestamp"": ""1693403881.273379"", ""content"": ""Please et me know what would be a good time to connect.""}, {""user"": ""aganivada"", ""timestamp"": ""1693404189.297019"", ""content"": ""now?""}, {""user"": ""aganivada"", ""timestamp"": ""1693405604.037519"", ""content"": ""<@U04JT69T00K> I started a collect job from cloudmapper locally, once the data is available we can generate report""}]" "1682507945.086169 ","[{""user"": ""aganivada"", ""timestamp"": ""1682507945.086169"", ""content"": ""<@U04JT69T00K> perimeter 81 seems to have some issue with the private dns resolution, if we configure both int and stage route53 resolvers resolution fails intermittently. Had a lengthy conv. with perimeter81 folks and they raised a ticket internally to check with their senior engineers. I will keep this thread posted on updates.""}, {""user"": ""aganivada"", ""timestamp"": ""1682606740.818729"", ""content"": "" Update on this:\n\nperimeter 81 folks say resolver should have mappings for both env's. Currently int resolver only resolves integration env's and stage resolver for stage env. They suggested us to make some changes from our side such that route53 resolver can respond to data from other account also. Or have 2 separate networks in perimeter 81 which would cost us about 7$/month but we will be able to apply granular rules on who can access stage.\n\nWe attempted to share the route 53 resolver from int to stage but coudn't get it working. <@U04JT69T00K> suggested an option to share route 53 hosted zone to , will attempt this in devx and int if it works then we can try the same between int and stage.\n\nIf we cant get this working I will setup an internal call to discuss if it makes sense to split int and stage into separate gateways in perimeter 81.""}, {""user"": ""aganivada"", ""timestamp"": ""1682677509.924409"", ""content"": ""Turns out there is a bug in perimeter81 dashboard new gateway will cost us USD 50 per month, working with AWS support to help resolve the issue.""}, {""user"": ""aganivada"", ""timestamp"": ""1683040996.861469"", ""content"": ""<@U026PMDB1ND> based on suggestions from AWS we need to setup VPC peering between int and stage this is pre-requisite for each side to be able to access resources on other side to help resolve DNS. Do you see any issues if we enabled peering between int and stage?\n\nMe and <@U04JT69T00K> established peering between devx and int and we were able to get cross account resolution working. Since CIDR ranges are unique for each of these env's we should not have any issue setting up between int and stage also.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683048071.766609"", ""content"": ""<@U02BV2DGUKC> - how much are we saving by doing this as opposed to keeping int and stage separate?""}, {""user"": ""aganivada"", ""timestamp"": ""1683048356.729249"", ""content"": ""<@U026PMDB1ND> if we opt for separating the gateway for stage that will cost us additional 50 dollars per month. Apart from traffic isolation one minor benefit with new gateway is that we can create a group that has granular access to stage via perimeter 81, today we are controlling it with AWS IAM.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683048485.477269"", ""content"": ""As of now, I am able to argue for either approach in my head :slightly_smiling_face:\n\nMaybe we can start with the peering approach and then add a gateway down the line? You see an issue with this?""}, {""user"": ""aganivada"", ""timestamp"": ""1683048617.113479"", ""content"": ""Sure Rama, that makes sense as of now we don't have a very strong reason for spending additional USD 50 on new gateway we can go with peering. If we have a use case then we can provision a gateway it should not be a big change.""}, {""user"": ""aganivada"", ""timestamp"": ""1683120224.795749"", ""content"": "" <@U04JT69T00K> please review how-to article on DNS config with P81 private DNS ""}]" "1676442504.806239 ","[{""user"": ""aganivada"", ""timestamp"": ""1676442504.806239"", ""content"": ""<@U0281D3GWHL> we are working on building metrics dashboard for internal consumption and one of the issues we are running into is to map id's to names for example tenant id, product instance id etc. Grafana has an ability to override id with an external value if we have the map (key-value pairs) for this we wanted to check if Dronahq has ability to filter and export mappings to a csv so we can add the mappings in Grafana board. cc: <@U03KLHDKL1H>""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1676442964.494599"", ""content"": ""<@U02BV2DGUKC> this is one of the problems that I am trying to solve in DronaHQ as well. My use case is that when I make an API call and get a tenant id in the result set, I wanted to show the tenant name in the UI and there was no easy way out. Just got some pointers that I am trying out to see if that helps. If that works, then yes, we can easily export out a CSV from the data table in the UI of DronaHQ. Hope that helps?""}, {""user"": ""aganivada"", ""timestamp"": ""1676443241.793139"", ""content"": ""sure thank you <@U0281D3GWHL>, sure it helps.... just one question lets say we had a table that just have tenant id and name somewhere in drona-hq will we be able to export it as a csv? just considering should we develop scripts for generating these mapping csv's or could we leverage dronahq UI to download the mappings""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1676443807.512219"", ""content"": ""Yes, we can download CSV's for any backend API. I just showed <@U03KLHDKL1H> what we can do as well""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1676443899.425209"", ""content"": ""Here is an example of CSV that I exported for all customers from INT.""}, {""user"": ""askumar"", ""timestamp"": ""1676444143.290399"", ""content"": ""Big thankyou <@U0281D3GWHL> this is exactly similar to script that we are writing.\n<@U02BV2DGUKC> the dashboard can give this output in 1 click for customers, vendors and product easily. We can leverage this, currently it is tied to INT env but with extended license we can map it to other ENV as well.""}, {""user"": ""aganivada"", ""timestamp"": ""1676445192.940239"", ""content"": ""cool thanks a lot <@U0281D3GWHL> when are we targeting to integrate with prod?""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1676448275.441889"", ""content"": ""Soon <@U02BV2DGUKC>, will keep you updated.""}, {""user"": ""aganivada"", ""timestamp"": ""1676448570.559009"", ""content"": ""Sure, thank you <@U0281D3GWHL>""}]" "1688999762.440219 ","[{""user"": ""aganivada"", ""timestamp"": ""1688999762.440219"", ""content"": ""<@U03KLHDKL1H> did we merge account-refresher to develop branch of job-executor? ""}, {""user"": ""askumar"", ""timestamp"": ""1688999888.391959"", ""content"": ""sorry <@U02BV2DGUKC> Not yet\nThere was one task pending on it for adding variables via cdk.\nI am executing it locally to avoid expiry""}, {""user"": ""aganivada"", ""timestamp"": ""1688999930.788359"", ""content"": ""ohh ok if it is a small task can we close it and merge the changes?""}, {""user"": ""aganivada"", ""timestamp"": ""1688999958.038489"", ""content"": ""I have some updates to this project so things might get tricky later on to merge""}, {""user"": ""askumar"", ""timestamp"": ""1688999983.247609"", ""content"": ""sure..will close it.""}, {""user"": ""askumar"", ""timestamp"": ""1689000805.753949"", ""content"": ""Thanks for catching this <@U02BV2DGUKC>, merged the changes, pipeline has passed.""}]" "1681233546.774449 ","[{""user"": ""aganivada"", ""timestamp"": ""1681233546.774449"", ""content"": ""<@U02SF36PVKL> can you share the collab ticket where we had pulse metrics product is looking for? cc: <@U03KLHDKL1H>""}, {""user"": ""mnirmal"", ""timestamp"": ""1681233594.681799"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1681233736.463319"", ""content"": ""thank you <@U02SF36PVKL>\n\ncc: <@U026PMDB1ND> for pulse metrics added to the ticket""}]" "1680542234.836589 ","[{""user"": ""aganivada"", ""timestamp"": ""1680542234.836589"", ""content"": ""<@U03NZ7Z52S2> can we attempt the flow again? we tried to make a direct call to put permissions with append mode and we see that it is working (adding product instead of replacing) cc: <@U02GC8SE18V> <@U0336QZAF98>""}, {""user"": ""aganivada"", ""timestamp"": ""1680545576.430329"", ""content"": ""<@U02GC8SE18V> in the flow where we send an invitation to a product admin via vendor user do we send empty permissions? like array with no permissions? or do we trigger invite member without any permissions in payload?\n\nI see there are 2 methods but not sure which one of them is getting invoked during the vendor inviting customer flow. cc: <@U0336QZAF98>""}, {""user"": ""rvaidya"", ""timestamp"": ""1680578734.350289"", ""content"": ""During Onboarding :: When vendor is inviting Customer : we are calling :\n```POST /users/invitation/{tenantId} on Tenancy (inviteUsersToOrg)```\nAnd we dont pass any permissions here.\n\n\ninviteUsersWithPermissions is called when collaborator is invites from collaborator section.\n\n\n<@U02BV2DGUKC> <@U0336QZAF98>""}, {""user"": ""rvaidya"", ""timestamp"": ""1680578817.584869"", ""content"": ""But its always been like this since day1 \u2026so this bug just got uncovered? <@U03NZ7Z52S2> <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1680579638.700249"", ""content"": ""<@U02GC8SE18V> this seems to be side effect of invite with permissions changes. When we we send empty permissions we are setting usermetadata as empty array instead we should not send usermetadata if the permissions are not passed. I am fixing this in tenancy cc: <@U0431DZTPJM> <@U0336QZAF98>""}, {""user"": ""aganivada"", ""timestamp"": ""1680579696.107719"", ""content"": ""this hasn't changed for at least a release so I think we have this issue currently but since we didnt touch this path we may not have observed this""}, {""user"": ""bganganna"", ""timestamp"": ""1680579739.240149"", ""content"": ""<@U02GC8SE18V> <@U02BV2DGUKC> I remember testing this flow during csm onboarding testing but not after collaborators related changes""}, {""user"": ""aganivada"", ""timestamp"": ""1680579805.421599"", ""content"": ""ack <@U03NZ7Z52S2> I think this flow regressed after we made changes for invite with permissions during collaborators flow""}, {""user"": ""rvaidya"", ""timestamp"": ""1680579880.770119"", ""content"": ""That makes sense <@U02BV2DGUKC>, and it also explains how it was working all this while.\nSo its a regression when we added inviteUserWithPermissions.\n\nThanks <@U03NZ7Z52S2> , <@U02BV2DGUKC> and <@U0336QZAF98>""}, {""user"": ""aganivada"", ""timestamp"": ""1680580257.608849"", ""content"": """"}, {""user"": ""rsrinivasan"", ""timestamp"": ""1680581770.968749"", ""content"": ""<@U02BV2DGUKC> - Can we raise auth0 ticket that there is no audit log in auth0 when user metadata is updated via invitation for existing user - this will help in future - even if via some flow - it got updated""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1680603035.058569"", ""content"": """"}, {""user"": ""rsrinivasan"", ""timestamp"": ""1680603051.861279"", ""content"": ""Here is auth0 ticket""}, {""user"": ""aganivada"", ""timestamp"": ""1680603214.932379"", ""content"": ""cool thank you <@U0336QZAF98>, missed your message on auth0 ticket earlier today""}]" "1689230921.402539 ","[{""user"": ""aganivada"", ""timestamp"": ""1689230921.402539"", ""content"": "" please review some FAQ's on multi-instance""}, {""user"": ""aganivada"", ""timestamp"": ""1689266513.599789"", ""content"": ""<@U0431DZTPJM> <@U04JT69T00K> please review when you get a chance if it looks good will share with rest of the teams.""}, {""user"": ""ppant"", ""timestamp"": ""1689315556.955279"", ""content"": ""LGTM <@U02BV2DGUKC>""}]" "1693540091.898339 ","[{""user"": ""aganivada"", ""timestamp"": ""1693540091.898339"", ""content"": ""<@U033PPLNFRU> I had a discussion with <@U02GC8SE18V> and <@U02HCMTQU3W> on handling flow where SSO admin assigns app to user without invitation. Technically it seems feasible but we wanted to check on when we are targeting the change. Considering we have a couple of new asks come up in last week (Intercom integration & platform changes to support new UI stack deployment) can we consider this change in 0.9.16?""}, {""user"": ""anair"", ""timestamp"": ""1693598378.003009"", ""content"": ""<@U02BV2DGUKC> could you elaborate on the flow please. Is it the same one as we previously discussed?""}, {""user"": ""aganivada"", ""timestamp"": ""1693629953.381329"", ""content"": ""yes <@U033PPLNFRU> this is pacden flow where admin added a user directly in IDP and they were seeing a blank page on login since org/role was not assigned to the user.""}]" "1693803512.798009 ","[{""user"": ""aganivada"", ""timestamp"": ""1693803512.798009"", ""content"": ""<@U03NZ7Z52S2> <@U03RSS0S76Y> I am getting error configuration exists when I try to configure pulse for platform tenant in our weekly retro tenant (axm_survey_stage). Not sure where the configuration exists, can you kindly take a look when you get a chance?""}, {""user"": ""bganganna"", ""timestamp"": ""1693811120.642629"", ""content"": ""<@U02BV2DGUKC> We had cancelled the instance , so the config was still active , so it dint allow to create the config again, <@U03RSS0S76Y> disabled config in dB, Can u please try again.""}, {""user"": ""aganivada"", ""timestamp"": ""1693812910.909989"", ""content"": ""sure let me do that""}, {""user"": ""aganivada"", ""timestamp"": ""1693813165.998669"", ""content"": ""it is working now, thank you <@U03NZ7Z52S2> and <@U03RSS0S76Y>\n\njust for my info, how do I cancel the config from UI?""}, {""user"": ""bganganna"", ""timestamp"": ""1693814467.717369"", ""content"": ""In the pulse list, if we can click on 3 dots , for recurring scheduled pulse there will be option to cancel the config or to stop the pulse series .""}, {""user"": ""snangia"", ""timestamp"": ""1693814547.906009"", ""content"": ""<@U02BV2DGUKC>, if we click on \""cancel\"", the instance goes into cancelled state and doesn't show up on UI so we cant modify the config anymore from the UI, there's an open tracker for this to modify the UX for cancelled instances.""}, {""user"": ""aganivada"", ""timestamp"": ""1693817211.450289"", ""content"": ""thank you <@U03RSS0S76Y>, can you share the tracker ticket ? we definitely need this to be addressed before customers start using scheduled pulses otherwise it is very confusing. cc: <@U02HCMTQU3W> ""}, {""user"": ""snangia"", ""timestamp"": ""1693817598.201579"", ""content"": ""cc: <@U03NZ7Z52S2> can u please share the tracker""}, {""user"": ""bganganna"", ""timestamp"": ""1693817718.887449"", ""content"": ""last one i could not find so i created new one . this was discussed as part of bug""}, {""user"": ""aganivada"", ""timestamp"": ""1693819042.992979"", ""content"": ""thank you <@U03NZ7Z52S2>""}, {""user"": ""aganivada"", ""timestamp"": ""1693819482.747379"", ""content"": ""<@U02GC8SE18V> /<@U04EV0M2VA6> jfyi, without it might be tricky to handle scheduled pulse configs. For context, we currently show an error stating there is an existing config which needs to be deleted before a new config can be added but we cant find the existing config in UI to delete.""}, {""user"": ""mli"", ""timestamp"": ""1693868137.948129"", ""content"": ""A couple of options here:\n1. If a Pulse instance is cancelled and it\u2019s the last scheduled instance in a series, then we can delete the config/parent series completely\n2. When users cancel a Pulse instance, we notify them that it\u2019s part of a recurring series and ask if they want to stop the entire Pulse series\n3. Once Pulses are cancelled, continue to show them in the app (e.g. in a section called) \u201cCancelled Pulses\u201d so that users can still view the questions and create a similar Pulse""}, {""user"": ""mli"", ""timestamp"": ""1693868213.482329"", ""content"": ""I also don\u2019t see the value in blocking users from creating new Pulses with the same configuration as an existing configuration. It\u2019s helpful to tell them: \u201cLook here, you have a Pulse with the same configuration. View that Pulse and then decide if you want to continue launching this new one\u201d but just blocking users from creating a similar series feels unintuitive""}]" "1688386664.908239 ","[{""user"": ""aganivada"", ""timestamp"": ""1688386664.908239"", ""content"": ""Hi Prabhu, why are we not linking other invitation types when sending invites? I had a case of admin inviting collabnorator so used ORGADMIN_INVITING_COLLABORATOR but it seems we are not sending invite in this case.""}, {""user"": ""ppant"", ""timestamp"": ""1688388132.985899"", ""content"": ""<@U02BV2DGUKC> We only added the templates in email-templates and related enums in tenancy . At that time we did not add the enums here in switch statement""}, {""user"": ""aganivada"", ""timestamp"": ""1688388333.772679"", ""content"": ""Ok thank you <@U0431DZTPJM> is it ok if we add some additional enums here or is there any pre-requisite before we add? Wanted to add one for orgadmin inviting collaborator""}, {""user"": ""ppant"", ""timestamp"": ""1688388372.680349"", ""content"": ""We can add here <@U02BV2DGUKC>, just pick up the template and work""}, {""user"": ""ppant"", ""timestamp"": ""1688388388.535729"", ""content"": ""Shall I do it?""}, {""user"": ""aganivada"", ""timestamp"": ""1688388527.520099"", ""content"": ""I am doing it for one of the template let me test it and then will add the rest""}, {""user"": ""aganivada"", ""timestamp"": ""1688389427.356459"", ""content"": ""<@U0431DZTPJM> can you review this error? \n\nis this because we dont have template uploaded to int?""}, {""user"": ""ppant"", ""timestamp"": ""1688389531.132879"", ""content"": ""I think there might be some issue with template""}, {""user"": ""ppant"", ""timestamp"": ""1688389548.573149"", ""content"": ""<@U02BV2DGUKC> Do you have a branch in which you are making changes? I can work on that if there is one""}, {""user"": ""aganivada"", ""timestamp"": ""1688389931.646469"", ""content"": ""for code I am working on tenancy feature/PLAT-2243, committed all my changes""}, {""user"": ""aganivada"", ""timestamp"": ""1688390257.456159"", ""content"": ""let me know once you are done will deploy and test the changes from the feature branch""}, {""user"": ""ppant"", ""timestamp"": ""1688391700.978689"", ""content"": ""<@U02BV2DGUKC> This should fix it. Had a stupid naming issue on the templates. Used _ instead of - :face_palm: ""}, {""user"": ""aganivada"", ""timestamp"": ""1688392328.758999"", ""content"": ""cool is it deplyed in int?""}, {""user"": ""aganivada"", ""timestamp"": ""1688392353.748079"", ""content"": ""can you also update deployment dependency page for 0.9.13?""}, {""user"": ""aganivada"", ""timestamp"": ""1688392767.540649"", ""content"": ""pushed the templates to int but still getting exception ""}, {""user"": ""ppant"", ""timestamp"": ""1688392940.989919"", ""content"": ""<@U02BV2DGUKC> Try now""}, {""user"": ""ppant"", ""timestamp"": ""1688392965.343999"", ""content"": ""Had another minor change in the template. Should work now""}, {""user"": ""aganivada"", ""timestamp"": ""1688392981.222819"", ""content"": ""Ok""}, {""user"": ""ppant"", ""timestamp"": ""1688393011.959879"", ""content"": ""Just tried sending email using this template, worked fine\n```curl -X 'POST' \\\n '' \\\n -H 'accept: application/json' \\\n -H 'Authorization: Bearer eyJraWQiOiJHUW9JcTh4Z3U5NFdGWCtyQ1BTcGJleHFwOGl6MmY5Y2diUUJVWjdpaDZVPSIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiI2bnU1Y2k1c2h1dXE3NjNtMG9va3QyZjVzNSIsInRva2VuX3VzZSI6ImFjY2VzcyIsInNjb3BlIjoiYXBpLWNhbGxzXC9yZXRyaWV2ZSBhcGktY2FsbHNcL3VwZGF0ZSBhcGktY2FsbHNcL2NyZWF0ZSBhcGktY2FsbHNcL2RlbGV0ZSIsImF1dGhfdGltZSI6MTY4ODM5MDY0NSwiaXNzIjoiaHR0cHM6XC9cL2NvZ25pdG8taWRwLnVzLXdlc3QtMi5hbWF6b25hd3MuY29tXC91cy13ZXN0LTJfWWZPU2oyNEdZIiwiZXhwIjoxNjg4Mzk3ODQ1LCJpYXQiOjE2ODgzOTA2NDUsInZlcnNpb24iOjIsImp0aSI6IjU3NDAzOWZmLWFhZDUtNGU3OC1hZWNmLTY1MDI3NjBkMGIzOSIsImNsaWVudF9pZCI6IjZudTVjaTVzaHV1cTc2M20wb29rdDJmNXM1In0.MDkvRtrkIIvxDegpzsY6t7m4--x8G2OMZGaFmp-B3H4UolVJ8JbLTw5L3KrlgDEjNlRedxizUNQcfB6Zugar-UWMDcHb_c_Pm1jJdjh48yd-1dxL2YO0cBub_hwq8yM5Q8VePg4W04UQfL6o82YbHWQb4z1iLn01Nrj3f1V2bdv5WgPBWDHo2SWG0u7246y2DYZKoH37NkGcT9lNFS-p-XnGJv3F2GmlgHhg-FMJ45_HbAJuabWVf_A2FjHeM_K1twQBvp9i3-_uh8wJXHxY0TexTRNXQphgAH0FmwWMjOBcv7aEyDYbchc49Z-Z56u_ghEKvEqhPKsVlg_8djqJzg' \\\n -H 'Content-Type: application/json' \\\n -d '{\n \""to\"": [\n {\n \""name\"": \""pp\"",\n \""email\"": \""\"",\n \""subject\"": \""string\"",\n \""templateInput\"": {\n \""inviter\"": \""YoInviter\"",\n \""org\"": \""YoOrg\"",\n \""invitee\"": \""YoInvitee\"",\n \""signupUrl\"": \""YoSignupUrl\""\n\n }\n }\n ],\n \""from\"": {\n \""name\"": \""PP\"",\n \""email\"": \""\""\n },\n \""subject\"": \""testing the new template\"",\n \""content\"": [\n {\n \""type\"": \""text/plain\"",\n \""data\"": \""string\"",\n \""location\"": \""string\""\n }\n ],\n \""templateName\"": \""tenancy-orgadmin-inviting-collaborator\"",\n \""category\"": \""ORGADMIN_INVITING_COLLABORATOR_EMAIL\""\n}'```""}, {""user"": ""aganivada"", ""timestamp"": ""1688393115.598439"", ""content"": ""cool, it worked but didn't seem to update the fields properly""}, {""user"": ""ppant"", ""timestamp"": ""1688393167.794129"", ""content"": ""<@U02BV2DGUKC> I think you missed adding some param""}, {""user"": ""aganivada"", ""timestamp"": ""1688393208.639359"", ""content"": ""hmmm ok let me check""}, {""user"": ""ppant"", ""timestamp"": ""1688393227.183889"", ""content"": """"}, {""user"": ""ppant"", ""timestamp"": ""1688393230.324589"", ""content"": ""Its because of this I guess""}, {""user"": ""ppant"", ""timestamp"": ""1688393255.988579"", ""content"": ""The invitation type here is correct but the part where it is being executed does not have the entire info""}, {""user"": ""aganivada"", ""timestamp"": ""1688393347.574509"", ""content"": ""hmmm..... ideally it should have it since the invitation is getting triggerred in auth0 properly""}, {""user"": ""ppant"", ""timestamp"": ""1688393434.905719"", ""content"": ""Ah ok, got the issue. The method where we are building the object for email template does not have all params""}, {""user"": ""ppant"", ""timestamp"": ""1688393443.860029"", ""content"": ""Based on the email type we should build this""}, {""user"": ""aganivada"", ""timestamp"": ""1688393524.819829"", ""content"": ""hmmm based on screenshot it should atleast have link and invitor name""}, {""user"": ""aganivada"", ""timestamp"": ""1688393605.807819"", ""content"": ""ok looks like the keys are different we are passing \""INVITATION_LINK_KEY\"" and template is looking for \""signupUrl\"" ?""}, {""user"": ""aganivada"", ""timestamp"": ""1688393652.013339"", ""content"": ""similarly invitor instead of inviter""}, {""user"": ""aganivada"", ""timestamp"": ""1688393724.439909"", ""content"": ""are you updating the method or shall I add these additional keys?""}, {""user"": ""ppant"", ""timestamp"": ""1688393745.410369"", ""content"": ""Give me a min, will do""}, {""user"": ""ppant"", ""timestamp"": ""1688394902.736409"", ""content"": ""<@U02BV2DGUKC> Currently we don\u2019t pass the invitee name in API, only emails. The new templates require the name also""}, {""user"": ""aganivada"", ""timestamp"": ""1688395144.859949"", ""content"": ""name of the invitee?""}, {""user"": ""ppant"", ""timestamp"": ""1688395154.528259"", ""content"": ""Yes""}, {""user"": ""ppant"", ""timestamp"": ""1688395158.324379"", ""content"": ""call?""}, {""user"": ""aganivada"", ""timestamp"": ""1688395173.184859"", ""content"": ""sure""}, {""user"": ""aganivada"", ""timestamp"": ""1688399475.398899"", ""content"": ""<@U0431DZTPJM> let me know once your changes are done will submit for add/patch permissions API for review cc: <@U02GC8SE18V> <@U042KRZPXHT>""}, {""user"": ""aganivada"", ""timestamp"": ""1688456656.228669"", ""content"": ""<@U0431DZTPJM> why are we mandating ssoconnection, there could be orgs which dont have sso enabled in this case we are returning error instead we could skip populating sso name""}, {""user"": ""ppant"", ""timestamp"": ""1688459036.567089"", ""content"": ""Oh, my bad. Initially I had placed this inside the switch block but then decided to keep it outside. Missed removing this line here""}, {""user"": ""aganivada"", ""timestamp"": ""1688459076.883069"", ""content"": ""Had to make a few more changes for tests to pass can you review when you get a chance?""}, {""user"": ""ppant"", ""timestamp"": ""1688459120.881849"", ""content"": ""<@U02BV2DGUKC> LGTM""}]" "1692823531.019669 ","[{""user"": ""anair"", ""timestamp"": ""1692823531.019669"", ""content"": ""<@U0336QZAF98> <@U02BV2DGUKC> did we make saml the default log in for pacden/17160. If not could we please do so?""}, {""user"": ""aganivada"", ""timestamp"": ""1692844760.744709"", ""content"": ""checking <@U033PPLNFRU>""}, {""user"": ""aganivada"", ""timestamp"": ""1692844922.265869"", ""content"": ""I see default primary connection set to saml flow for pacden (17160)""}, {""user"": ""aganivada"", ""timestamp"": ""1692845377.506019"", ""content"": ""looks like has not been removed from username password after saml flow was enabled""}, {""user"": ""anair"", ""timestamp"": ""1692845438.941119"", ""content"": ""<@U02BV2DGUKC> <@U0336QZAF98> what is with the ff? It seems we did not turn it on and both EA and PacDen onboarded without SSO. Can we move dependency from ff and do it via code""}, {""user"": ""anair"", ""timestamp"": ""1692845564.504529"", ""content"": ""*new users at EA and Pacden""}, {""user"": ""aganivada"", ""timestamp"": ""1692845564.739249"", ""content"": ""this is not a ff <@U033PPLNFRU> we make this call via API. once this config is set all new users should get invitations via SAML flow, unless the old user(username-password) has not been deleted from auth0. Are we noticing a different behaviour?""}, {""user"": ""anair"", ""timestamp"": ""1692845593.252259"", ""content"": ""new users at EA and pacden were not invited via SSO even though they were invited on the UI""}, {""user"": ""aganivada"", ""timestamp"": ""1692845615.825159"", ""content"": ""ok let me check""}, {""user"": ""aganivada"", ""timestamp"": ""1692846092.165329"", ""content"": ""<@U0336QZAF98> do you see any issue with enabling for all tenants in prod? looks like it is enabled for select tenants even in stage""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1692846125.929889"", ""content"": ""No <@U02BV2DGUKC> - we can enable it""}, {""user"": ""aganivada"", ""timestamp"": ""1692846168.352849"", ""content"": ""will first enable on stage""}, {""user"": ""aganivada"", ""timestamp"": ""1692848619.828979"", ""content"": ""looks like ff is the issue <@U033PPLNFRU>, I think we missed turning on the flag during onboarding saml steps for EA and pacden. in Stage onboarding weems to be working after turning on the flag, will run a few more tests and enable flag for all tenants""}, {""user"": ""anair"", ""timestamp"": ""1692848659.860449"", ""content"": ""thanks <@U02BV2DGUKC> <@U0336QZAF98> please keep me updated""}, {""user"": ""aganivada"", ""timestamp"": ""1692853236.120659"", ""content"": ""<@U03NZ7Z52S2> / <@U03DHUAJVMK> can you help with trying admins invitation flow in stage? we turned on a flag related to sso flow to enable routing invitations through saml only for tenants where primary connection is saml, for other tenants regular username password flow should trigger. I tested with a few users flow seems to be working, since we are planning to enable this in prod it will great if you can take a look as well.\n\nI enabled SAML flow as default for our platform retro tenant for testing so any invitation sent from the axm-survey-stage tenant will go via okta admin creds to addd user and app are \n\n/Axiamatic@12345""}, {""user"": ""bot_message"", ""timestamp"": ""1692859973.950399"", ""content"": ""@Anu created a Task TEST-1100 Admin invitation flow testing""}, {""user"": ""bot_message"", ""timestamp"": ""1692859987.505589"", ""content"": ""@Anu assigned TEST-1100 Admin invitation flow testing from @Anu \u27f6 bhavana""}, {""user"": ""bganganna"", ""timestamp"": ""1692866439.206149"", ""content"": ""<@U02BV2DGUKC> I m seeing error=access_denied&error_description=the%20specified%20account%20is%20not%20allowed%20to%20accept%20the%20current%20invitation&state=UjhrTUtyWE9sQ0Q4LTUwS3cuMUhLMlZaV1BfT1dkN0ZjREpZUX5Zci1iZw%3D%3D post accepting the invite , not sure if i did something wrong.""}, {""user"": ""aganivada"", ""timestamp"": ""1692869937.524169"", ""content"": ""<@U03NZ7Z52S2> can you share more details on which user was invited to which org? this usually happens if we invite a user on an org where connection is not enabled""}, {""user"": ""bganganna"", ""timestamp"": ""1692875170.985919"", ""content"": ""<@U02BV2DGUKC> Completed validation with and without saml connection by inviting user from admins page and adding the user as PO for an engagement .""}, {""user"": ""aganivada"", ""timestamp"": ""1692875287.180679"", ""content"": ""awesome thanks a lot <@U03NZ7Z52S2> , will keep the FF turned on in stage. Please let me know if we run into any issues. And as discussed lets see if we can use at least one tenant in stage with SAML since all major customer setups are using SAML flow cc: <@U03DHUAJVMK>""}, {""user"": ""bganganna"", ""timestamp"": ""1692875354.206579"", ""content"": ""Sure <@U02BV2DGUKC> can we use our retro tenant for SAML testing ?""}, {""user"": ""aganivada"", ""timestamp"": ""1692875515.416469"", ""content"": ""<@U033PPLNFRU> <@U0336QZAF98> <@U026PMDB1ND> setting tenancy.security-settings-sso-invitation turned on for all tenants in prod.\n\n<@U026PMDB1ND> do you recollect why we set ticket to \""NOT REQUIRED\""? we can pick it up in 0.9.15 if it makes sense""}]" "1676993223.662999 ","[{""user"": ""pjha"", ""timestamp"": ""1676993223.662999"", ""content"": ""<@U02BV2DGUKC> <@U026PMDB1ND> Please review ""}, {""user"": ""aganivada"", ""timestamp"": ""1676993926.627669"", ""content"": ""thank you <@U04JT69T00K> added some comments can you kindly check?""}, {""user"": ""pjha"", ""timestamp"": ""1677042609.017419"", ""content"": ""<@U02BV2DGUKC> I have replied to the questions.""}]" "1692848933.959189 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1692848933.959189"", ""content"": ""Just fyi: The shout-out message was generated by taking the kudos messages in the thread, sanitizing them a bit, running that whole thing through ChatGPT, and then editing it again to reduce some repetition. Human -> machine -> human pipeline :slightly_smiling_face:""}, {""user"": ""araman"", ""timestamp"": ""1692855160.357329"", ""content"": ""ChatGPT (as well) took QA or systest for granted and left us anonymous :slightly_smiling_face:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692895905.667439"", ""content"": "":slightly_smiling_face: BadGPT, <@U03DHUAJVMK>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692895975.640899"", ""content"": ""OTOH, Ask-AI did the right thing and highlighted the systest team in the retro summary. I guess as in life, even with AI we win some and we lose some :slightly_smiling_face:""}]" "1676446062.081179 ","[{""user"": ""araman"", ""timestamp"": ""1676446062.081179"", ""content"": ""Trivial, Medium, Hard, Rare, On-Call, Automation sounds good <@U026PMDB1ND>""}, {""user"": ""araman"", ""timestamp"": ""1677330720.922979"", ""content"": ""<@U03NZ7Z52S2> FYI""}]" "1689831702.731509 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1689831702.731509"", ""content"": "" - anyone able to see the ECS panel in AWS console of INT?""}, {""user"": ""aganivada"", ""timestamp"": ""1689831747.026459"", ""content"": ""yes <@U026PMDB1ND>""}, {""user"": ""aganivada"", ""timestamp"": ""1689831767.945309"", ""content"": """"}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689831809.462669"", ""content"": ""<@U02BV2DGUKC> can you paste the URL from the first screen shot""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689831836.463949"", ""content"": ""And try this URL: ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689831862.011669"", ""content"": ""I guess I selected v2 UI a long time back""}, {""user"": ""aganivada"", ""timestamp"": ""1689831876.882559"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1689831901.015669"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1689831907.280809"", ""content"": ""url works for me Rama""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689831922.255729"", ""content"": ""Something is off for me:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689831936.049649"", ""content"": ""Nice white page :slightly_smiling_face:""}, {""user"": ""aganivada"", ""timestamp"": ""1689831998.431409"", ""content"": ""weird.... is this the case even in incognito? wondering if vpn is causing this""}, {""user"": ""aganivada"", ""timestamp"": ""1689832006.386489"", ""content"": ""BTW I am on vpn""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689832074.437019"", ""content"": ""Looks like a Safari problem""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689832082.324829"", ""content"": ""I can see the page on Chrome""}]" "1679422220.190789 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1679422220.190789"", ""content"": ""Gist is so impressive:\n\n> <#C02CREJ9EJ2|platform-engg>**\n> The group is discussing the use of specific libraries for their jira lambda integration with VPC. Guru suggested refactoring the common methods into a plat-commons library, and then Prabhu Pant proposed renaming plat-libs-epoch to plat-libs-lambda. Rama suggested renaming it to plat-libs-py-common, as it could be used in non-lambda cases as well. Sanjiv Ranjan proposed making a tracker ticket for the changes.\n\n> **\n> Anil and Prabhu Pant discussed the logging occurring for tenant 19190 and found that the Ingestion Service was producing lots of debug logs. Anil suggested that the log level be reverted to info and Sagarika Nangia agreed to do it the next day. Anil also suggested that Paran should use the tenant name resolver API instead of making multiple API calls.\n\n> **\n> Anil asked Prashant about his progress on researching a load-balancer, to which Prashant replied that he was stuck with other issues related to setting up multiple instances but has now fixed them and will begin looking into the load-balancer.\n\n> **\n> Anil and Ashwani are discussing their attempts toOverride the context for their transaction manager. They have not succeded yet.\n\n> **\n> Anil and Rama discussed the possibility of having a LoadBalancer (LB) instead of IP addresses in Route53, comparing the number of requests made to LB versus the requests in the access log of tenancy. They found that more requests are made directly to the tenancy which suggests requests are not going through the load balancer. They agreed to register the LB with the service name, update CDK and deploy all services once to switch from xxx to xxx-lb. Anil found an article that discussed setting up AWS alias DNS names which would route traffic to an Elastic Load Balancing load balancer.\n\n> **\n> Prashant Jha shared a link about auto-deploying docker images to ECR, and Rama suggested further discussing how it works and how to properly implement it.\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679524512.106859"", ""content"": ""<@U034RLJA97X> - did we create a tracker for point# 1 above?\n\n<@U03BPNY5AGM> - Re: <@U02BV2DGUKC>'s suggestion of\n\n>> Anil also suggested that Paran should use the tenant name resolver API instead of making multiple API calls.\nAre we tracking this? Is this something we'd do soon?""}, {""user"": ""pkarthikeyan"", ""timestamp"": ""1679544456.717489"", ""content"": ""<@U026PMDB1ND> After an analysis I decided to stick with the core-data api as i found a flow where publisher id is made available, So either way there was only 1 hop, so decided to stay with publisher name from core-data.""}, {""user"": ""aganivada"", ""timestamp"": ""1679544509.516519"", ""content"": ""<@U03BPNY5AGM> that makes sense we take tenant name also from publisher table""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679544532.205599"", ""content"": ""Since publisher names don\u2019t change, can we cache them locally? <@U028EDANJM9> does it make searching to cache them?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679544569.039799"", ""content"": ""only we keep calling the api heavily ""}, {""user"": ""gshenoy"", ""timestamp"": ""1679544883.420769"", ""content"": ""Yes <@U026PMDB1ND>, publisher names are not likely to change. Makes sense to cache them.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679545241.574589"", ""content"": ""> does it make searching to cache them?\nThat was an interesting typo :slightly_smiling_face:""}, {""user"": ""sranjan"", ""timestamp"": ""1679579014.556739"", ""content"": ""<@U026PMDB1ND> created a tracker PLAT ticket - . cc <@U0431DZTPJM>""}, {""user"": ""ppant"", ""timestamp"": ""1679579344.622969"", ""content"": ""<@U026PMDB1ND> <@U034RLJA97X> Using this ticket to integrate all the services used by lambda. We can refer to PLAT-1840 when we rename `plat-libs-epoch` to `plat-libs-py-commons`\ncc <@U02BV2DGUKC>""}]" "1690345415.964489 ","[{""user"": ""askumar"", ""timestamp"": ""1690345415.964489"", ""content"": ""\nPlease take a look at the proposal doc for Auth0 User permission replication in Core\n""}, {""user"": ""ppant"", ""timestamp"": ""1690346368.158009"", ""content"": ""<@U03KLHDKL1H> Added a few questions""}, {""user"": ""askumar"", ""timestamp"": ""1690385789.019429"", ""content"": ""Thanks <@U0431DZTPJM> for the offline discussion, have updated the doc to simplify data storage structures.""}, {""user"": ""aganivada"", ""timestamp"": ""1690386086.658679"", ""content"": ""sorry <@U03KLHDKL1H> was occupied in some cxo dashboard work will review this by tmrw morning""}, {""user"": ""askumar"", ""timestamp"": ""1690386112.547249"", ""content"": ""no problem <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1690458374.857299"", ""content"": ""<@U03KLHDKL1H> added some inline and comments to the document PTAL overall it needs a few changes before we can open it up for review. most importantly lets switch to design template and try to fill as many sections as we can, for reference please review some articles under in folder \n\nFor db relation representation please check if we can use puml diagram as shown here it might also make sense to add a sequence diagram with puml to explain the flow there are some puml flows under tenancy > arch folder for reference. For component diagrams we can also use ""}, {""user"": ""aganivada"", ""timestamp"": ""1690458425.624309"", ""content"": ""please move the document under specs folder ""}, {""user"": ""askumar"", ""timestamp"": ""1690458488.219089"", ""content"": ""Sure... Thankyou <@U02BV2DGUKC>, will make the changes""}, {""user"": ""askumar"", ""timestamp"": ""1691087494.504389"", ""content"": ""<@U02BV2DGUKC> have addressed the comments, would get in touch for review and further changes.""}]" "1677702192.166039 ","[{""user"": ""svummidi"", ""timestamp"": ""1677702192.166039"", ""content"": ""<@U02BV2DGUKC> - Is there any way to get the tenant selection page after entering a tenant without logging out and logging in?""}, {""user"": ""aganivada"", ""timestamp"": ""1677728221.383799"", ""content"": ""not sure about how we can do that but should be possible <@U040RCBPBEC> with some UI effort. This flow/route is coded into our UI logic""}]" "1686586476.112839 ","[{""user"": ""snangia"", ""timestamp"": ""1686586476.112839"", ""content"": "" user-action service is stuck in update_rollback_in_progress since quite a while. Can someone please take a look?\n""}, {""user"": ""askumar"", ""timestamp"": ""1686588507.880349"", ""content"": ""\nHope you have the latest develop pulled <@U03RSS0S76Y>""}, {""user"": ""askumar"", ""timestamp"": ""1686588602.493459"", ""content"": ""Looks like a bean creation failure :\n""}, {""user"": ""askumar"", ""timestamp"": ""1686588694.888839"", ""content"": ""Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'axmTenantUserActionDelegate' defined in file [/opt/axm/com/axm/collaboration/user/action/service/AxmTenantUserActionDelegate.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'tenantAwareUserActionService' defined in file [/opt/axm/com/axm/collaboration/user/action/service/TenantAwareUserActionService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception\n\nIs this a recent addition <@U03RSS0S76Y> ?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686588752.251769"", ""content"": ""Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.redis.core.ReactiveRedisTemplate]: Factory method 'reactiveRedisTemplate' threw exception; nested exception is org.redisson.client.RedisConnectionException: Can't connect to servers!""}, {""user"": ""snangia"", ""timestamp"": ""1686588796.196909"", ""content"": ""<@U03KLHDKL1H> I didnt have the latest develop, that's why cancelled update""}, {""user"": ""snangia"", ""timestamp"": ""1686588803.950029"", ""content"": ""but rollback itself is not getting completed""}, {""user"": ""askumar"", ""timestamp"": ""1686588804.340739"", ""content"": ""<@U028EDANJM9> could you please have a look ?\nError creating bean with name 'personaRedisOperationsService' defined in URL [jar:file:/opt/axm/lib/core-persona-provider-0.0.2.jar!/com/axm/core/persona/provider/client/PersonaRedisOperationsService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'reactiveRedisTemplate' defined in class path resource [com/axm/core/persona/provider/config/PersonaCacheConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.redis.core.ReactiveRedisTemplate]: Factory method 'reactiveRedisTemplate' threw exception; nested exception is org.redisson.client.RedisConnectionException:""}, {""user"": ""snangia"", ""timestamp"": ""1686588845.765589"", ""content"": ""<@U03KLHDKL1H> this error will go away with latest cdk develop branch. Can you please help me with the rollback of stack.""}, {""user"": ""askumar"", ""timestamp"": ""1686588946.638289"", ""content"": ""I suppose we can remove it.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686588952.451289"", ""content"": ""Just set the desired count to 0, which stops the existing services and then deploy""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686588985.647559"", ""content"": ""```./desired-count.sh <profile> ../user-action-service 0```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686588993.995159"", ""content"": ""In apps/scripts""}, {""user"": ""snangia"", ""timestamp"": ""1686588996.688229"", ""content"": ""thanks <@U026PMDB1ND>""}, {""user"": ""askumar"", ""timestamp"": ""1686589031.160819"", ""content"": ""Did not know that ..thanks <@U026PMDB1ND> !""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686589293.066119"", ""content"": ""<@U03RSS0S76Y> - to get into this stage, did you go to the CF page on AWS console and cancel the deployment using \""Cancel Update Task\"" as described here: \n\nTrying to see if there's a cleaner way to get out than the desired-count approach.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686589411.754419"", ""content"": ""<@U03KLHDKL1H> - while deleting stack is an option it cleans up everything including the LB and the time taken to delete + deploy would easily be 30-40 minutes. The method of updating the number of desired tasks to 0 just reaps the existing tasks and is much faster (3-5 minutes) and doesn't touch any other part of the infra.""}, {""user"": ""askumar"", ""timestamp"": ""1686589525.054499"", ""content"": ""Got it !!""}, {""user"": ""snangia"", ""timestamp"": ""1686589549.145419"", ""content"": ""yes <@U026PMDB1ND>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686589760.532389"", ""content"": ""This looks like a \""clean hack\"" in such situations:""}, {""user"": ""snangia"", ""timestamp"": ""1686589812.075379"", ""content"": ""<@U026PMDB1ND> it's still stuck in same state:\n```ack Deployments Failed: ValidationError: Stack:arn:aws:cloudformation:us-west-2:433798924509:stack/user-action-stack-int/94242c20-03d5-11ee-b07f-02e1cbbbc16b is in UPDATE_ROLLBACK_IN_PROGRESS state and can not be updated.```""}, {""user"": ""snangia"", ""timestamp"": ""1686589823.316499"", ""content"": ""./desired-count.sh default ../user-action-service 0""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686589840.713899"", ""content"": ""Ok, let me take a look""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686589907.147799"", ""content"": ""Which version are you trying to deploy?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686589916.696419"", ""content"": ""\""desiredCount\"": 0,\n \""runningCount\"": 0,\n \""pendingCount\"": 0,""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686589922.124959"", ""content"": ""So the service is down""}, {""user"": ""snangia"", ""timestamp"": ""1686589931.515099"", ""content"": ""0.0.17-COLL-2545-SNAPSHOT""}, {""user"": ""snangia"", ""timestamp"": ""1686590179.705059"", ""content"": ""message-broker is also stuck in same state""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686590199.417229"", ""content"": ""Yeah.. I see 4 services in that state""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686590222.527699"", ""content"": ""Wonder if the update didn't go as planned, <@U04JT69T00K>""}, {""user"": ""snangia"", ""timestamp"": ""1686590225.997879"", ""content"": ""i see rollback completed now for user action and message-broker""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686590233.557399"", ""content"": ""okay""}, {""user"": ""snangia"", ""timestamp"": ""1686590252.485159"", ""content"": ""let me try to deploy them""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686590257.302029"", ""content"": ""Okay""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686590272.548819"", ""content"": ""Must have just timed out""}, {""user"": ""snangia"", ""timestamp"": ""1686590292.238099"", ""content"": ""yes could be""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686590333.468569"", ""content"": ""<@U04JT69T00K>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686590340.630739"", ""content"": ""fyi <@U040RCBPBEC>""}, {""user"": ""svummidi"", ""timestamp"": ""1686626024.766989"", ""content"": ""<@U0336QZAF98> there are no deployment issues but to test deployment, I used develop builds. There are lots of exceptions related to parsing notification events. Please check.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1686626831.943279"", ""content"": ""<@U040RCBPBEC> - i looked at the cases where exception is happening . Pulse Instance has finished and we are trying to get overallscore for tenantId and producttInsanceId - we have empty score from druid - so we are trying 3 times at 2 minutes interval -\n\n```curl -X 'GET' \\\n '' \\\n -H 'accept: application/json' \\\n -H 'Authorization: Bearer eyJraWQiOiJHUW9JcTh4Z3U5NFdGWCtyQ1BTcGJleHFwOGl6MmY5Y2diUUJVWjdpaDZVPSIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiI2bnU1Y2k1c2h1dXE3NjNtMG9va3QyZjVzNSIsInRva2VuX3VzZSI6ImFjY2VzcyIsInNjb3BlIjoiYXBpLWNhbGxzXC9yZXRyaWV2ZSBhcGktY2FsbHNcL3VwZGF0ZSBhcGktY2FsbHNcL2NyZWF0ZSBhcGktY2FsbHNcL2RlbGV0ZSIsImF1dGhfdGltZSI6MTY4NjYyNjU2NywiaXNzIjoiaHR0cHM6XC9cL2NvZ25pdG8taWRwLnVzLXdlc3QtMi5hbWF6b25hd3MuY29tXC91cy13ZXN0LTJfWWZPU2oyNEdZIiwiZXhwIjoxNjg2NjMzNzY3LCJpYXQiOjE2ODY2MjY1NjcsInZlcnNpb24iOjIsImp0aSI6IjNmMGE3ZDcwLTY4ZjktNGY2Ny04MWYxLTk5MTAyNzg1OTVmMyIsImNsaWVudF9pZCI6IjZudTVjaTVzaHV1cTc2M20wb29rdDJmNXM1In0.d4u7NgVpydVjUQWqDggKNBEh-DRza_B_P88MHwfp-RR1IxV8gpsMs4_h8jd4nJUluwKBoQ4KMQvc9d83_DmsSoxmYzgDGBUBhAMY2b3uzLF5V2LWGBJ4DqszzM4t9tdaFbHJfahle3FD0RDSGYaWGYwgJRvcHjw8J74FGuaSq_0ksuFylvzayBF1wrLVzwk2BZWs5VGlMHC--AL7Pv-XF5Q6cN4hjgukH2wWQebSAhgXGkI3gPsb_lZJJf8zNY8EF13K22_n42aItV2qAv3wKMztPgPjjHTv8IbUrISRUZVhf7v1gQvaSUBBB2miUjbZ5RqH0ZvpVSETH0AzyKXtdA'\n\nResponse:\n{\n \""data\"": {\n \""aggType\"": \""year\"",\n \""point\"": null,\n \""pointDiff\"": null\n },\n \""success\"": true,\n \""error\"": null\n}```""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1686627702.494379"", ""content"": ""In the last 12 hours , for the tenant and product instance combination , we have empty scores in druid\t\n\ttenant\tproduct instance\n\t33606\t44701\n\t19190\t13384\n\t27137\t17028\n\t41810\t35583\n\t51241\t35720\n\t49370\t37747\n\t49370\t30541""}, {""user"": ""pjha"", ""timestamp"": ""1686654044.642409"", ""content"": ""<@U026PMDB1ND> should not be because of the update. I will check it once again""}, {""user"": ""svummidi"", ""timestamp"": ""1686722070.057239"", ""content"": ""<@U0336QZAF98> - Can we add checks in the code to prevent these \u201cjava.lang.IllegalStateException\u201d - If it is an expected case, we need to handle that condition and log info or warn log instead of stack trace.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1686753229.334849"", ""content"": ""<@U040RCBPBEC> - Can we sync up on this ?""}]" "1683309101.999949 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1683309101.999949"", ""content"": ""<@U02BV2DGUKC> <@U03KLHDKL1H> <@U04JT69T00K> - can we please switch off the RDS/Aurora instances in INT that are not being used. RDS tends to suck in a lot of $s if we are not careful.""}, {""user"": ""askumar"", ""timestamp"": ""1683311074.093089"", ""content"": ""Sure <@U026PMDB1ND> , will turn them off""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683647371.710079"", ""content"": ""<@U03KLHDKL1H> - do we need all of these:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683647390.361439"", ""content"": ""Also the naming of the two aurora clusters seems to be not following any pattern""}, {""user"": ""askumar"", ""timestamp"": ""1683647590.406779"", ""content"": ""Hi <@U026PMDB1ND>\naurora-aslv2-common-int ..we have created for actual usage in INT\nIn production name was like aslv2-common-production , will check with <@U04JT69T00K> to review it.\n\nCommons cluster is just a test cluster that\nI am using to test any dumps and scripts, I stopped it temporarily over weekend, activated again for testing, it will be closed soon.""}, {""user"": ""askumar"", ""timestamp"": ""1683648398.967909"", ""content"": ""About the other instances, we will stop the instances not being used, keep them for couple of days and delete it within this week.""}, {""user"": ""aganivada"", ""timestamp"": ""1683648925.422189"", ""content"": ""<@U03KLHDKL1H> can we also check with Nayan (or anyone involved) if we want the replication postgres one?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683648971.237569"", ""content"": ""> I am using to test any dumps and scripts, I stopped it temporarily over weekend, activated again for testing, it will be closed soon.\nThanks!""}, {""user"": ""askumar"", ""timestamp"": ""1683648994.252199"", ""content"": ""sure <@U02BV2DGUKC> , will confirm with him/collab.""}, {""user"": ""askumar"", ""timestamp"": ""1683649385.988449"", ""content"": ""<@U026PMDB1ND> <@U02BV2DGUKC>\nFor the cluster name in INT and STAGE\nwe would keep them as *aslv2-common-int and aslv2-common-stage*\non lines of production names.""}, {""user"": ""askumar"", ""timestamp"": ""1683649395.286099"", ""content"": ""cc <@U04JT69T00K>""}]" "1684148624.957249 ","[{""user"": ""aganivada"", ""timestamp"": ""1684148624.957249"", ""content"": ""<@U026PMDB1ND>Tried deploying lambda's to int with cdkv2 most of deployments worked. there was one minor import issue with coordination building blocks .\n\n<@U04JT69T00K> found an issue while deploying trigger eventbridge changes synth works fine but cdk throws error during deployment. He is attempting to fix this from CDK. Prashant, please post summary of the error we saw during deployment.""}, {""user"": ""pjha"", ""timestamp"": ""1684150413.032579"", ""content"": ""synth.txt""}, {""user"": ""pjha"", ""timestamp"": ""1684156507.719589"", ""content"": ""<@U026PMDB1ND> please review ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684168614.940979"", ""content"": ""Looks good, <@U04JT69T00K>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684168637.640549"", ""content"": ""Thanks <@U04JT69T00K> and <@U02BV2DGUKC>\n\nCan we merge into develop now?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684168652.440379"", ""content"": ""I'd like us to deploy 9.11 to stage using v2 code""}, {""user"": ""aganivada"", ""timestamp"": ""1684168720.797209"", ""content"": ""Yes <@U026PMDB1ND> I think it is better to merge. We will watch out for any issues during this week before we deploy to stage.""}]" "1689290415.384039 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1689290415.384039"", ""content"": ""<@U02BV2DGUKC> - following up on the junk in prod-db discussion from the platform retro:\n\n```postgresdbproduction=> select count(*) from publisher where name like 'publisher-%';\n count\n-------\n 4416\n(1 row)\n\npostgresdbproduction=> select count(*) from product_instance where name like 'product-%';\n count\n-------\n 1970\n(1 row)\n\npostgresdbproduction=> select count(*) from product where name like 'product-%';\n count\n-------\n 3925\n(1 row)\n\npostgresdbproduction=> select count(*) from product_category where name like 'category-%';\n count\n-------\n 744\n(1 row)```\nLooks like we haven't left behind any more junk after June 23rd (at least in these 4 tables)\n\n```postgresdbproduction=> select max(created_at) from product_category where name like 'category-%';\n max\n-------------------------------\n 2023-06-23 07:05:26.934554+00\n(1 row)\n\npostgresdbproduction=> select max(created_at) from product where name like 'product-%';\n max\n-------------------------------\n 2023-06-19 04:10:31.680907+00\n(1 row)\n\npostgresdbproduction=> select max(created_at) from product_instance where name like 'product-%';\n max\n-------------------------------\n 2023-06-18 05:16:33.227221+00\n(1 row)\n\npostgresdbproduction=> select max(created_at) from publisher where name like 'publisher-%';\n max\n-------------------------------\n 2023-06-23 07:05:30.939308+00\n(1 row)```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689290523.039279"", ""content"": ""I don't think we should invoke tenant deletion flow for 1000s. We have 1994 tenants corresponding to the 4.4k junk publishers.\n\nI think we should just run scripts to clear them up.""}, {""user"": ""aganivada"", ""timestamp"": ""1689305520.891509"", ""content"": ""ack <@U026PMDB1ND>""}, {""user"": ""bganganna"", ""timestamp"": ""1689308253.350419"", ""content"": ""<@U02BV2DGUKC> can we use the script which we ran on stage .""}, {""user"": ""aganivada"", ""timestamp"": ""1689308389.355779"", ""content"": ""sure <@U03NZ7Z52S2>, will schedule a call to go over this""}, {""user"": ""bganganna"", ""timestamp"": ""1689316811.234819"", ""content"": ""<@U02SF36PVKL> <@U02D4DUKDQC> Can we check if there are any stale pulses in prod as we are planning to cleanup the publishers and products. If any stale PVT's exists in PMS it might cause some alerts cc <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1689571197.924529"", ""content"": ""<@U02SF36PVKL> did we get a chance to look into this? Basically we want to be sure that running a PVT/product cleanup for stale entries will not trigger any unexpected alerts in prod.""}, {""user"": ""mnirmal"", ""timestamp"": ""1689571933.152459"", ""content"": ""Yeah <@U02BV2DGUKC>, we will need to delete it for PMS too. <@U03NZ7Z52S2> can we delete it for all the systest tenants? We don't have any recurring configs for any sys test tenants right?""}, {""user"": ""bganganna"", ""timestamp"": ""1689572624.351789"", ""content"": ""We have removed from systests, but its better to check once in prod <@U02SF36PVKL>""}, {""user"": ""bganganna"", ""timestamp"": ""1689572726.253279"", ""content"": ""<@U02BV2DGUKC> can u please share the TID for\n```%{APPS_TENANT_ID_TEAMS}, %{APPS_TENANT_ID_SLACK}, %{APPS_TENANT_ID_WEBAPP_ONLY} ```\n<@U02SF36PVKL> we can check only for these TID's for now""}, {""user"": ""aganivada"", ""timestamp"": ""1689572828.076579"", ""content"": ""ack""}, {""user"": ""aganivada"", ""timestamp"": ""1689572936.773089"", ""content"": """"}, {""user"": ""mnirmal"", ""timestamp"": ""1689574995.881079"", ""content"": ""```SELECT * FROM svc_pulse_manager_default_axm.pulse_survey ps inner join svc_pulse_manager_default_axm.pulse_survey_config psc on ps.pulse_survey_config_id = psc.id where ps.tenant_id in (9615,1932,1933) where ps.status='ACTIVE';```\n""}, {""user"": ""mnirmal"", ""timestamp"": ""1689575022.844009"", ""content"": ""```SELECT * FROM svc_pulse_manager_default_axm.pulse_survey ps inner join svc_pulse_manager_default_axm.pulse_survey_config psc on ps.pulse_survey_config_id = psc.id where ps.tenant_id in (9615,1932,1933) where ps.status='DELETED';```\n""}, {""user"": ""mnirmal"", ""timestamp"": ""1689575045.031859"", ""content"": ""<@U03NZ7Z52S2> I think we can get rid of the soft deleted entries also from the prod. WDYT?""}, {""user"": ""mnirmal"", ""timestamp"": ""1689575073.631789"", ""content"": ""<@U02BV2DGUKC> can you please share the result of the 2 queries mentioned above?""}, {""user"": ""bganganna"", ""timestamp"": ""1689575426.174839"", ""content"": ""We can but its better if we do it later once this is done <@U02SF36PVKL>""}, {""user"": ""aganivada"", ""timestamp"": ""1689577001.662889"", ""content"": ""<@U02SF36PVKL> fixed above queries to\n\nSELECT * FROM svc_pulse_manager_default_axm.pulse_survey ps inner join svc_pulse_manager_default_axm.pulse_survey_config psc on ps.pulse_survey_config_id = psc.id where ps.tenant_id in (9615,1932,1933) and ps.state='ACTIVE';\n\n\nSELECT * FROM svc_pulse_manager_default_axm.pulse_survey ps inner join svc_pulse_manager_default_axm.pulse_survey_config psc on ps.pulse_survey_config_id = psc.id where ps.tenant_id in (9615,1932,1933) and ps.state='DELETED';\n\n\nActive => 7 results\nDeleted => 3300""}, {""user"": ""bganganna"", ""timestamp"": ""1689577128.534459"", ""content"": ""<@U02SF36PVKL> If there are only 7PVT's are active then we can execute the delete survey via pvtID from swagger itself right?""}, {""user"": ""aganivada"", ""timestamp"": ""1689577280.759179"", ""content"": ""<@U03NZ7Z52S2> stage cleanup is done ""}, {""user"": ""mnirmal"", ""timestamp"": ""1689577315.409249"", ""content"": ""yeah we can <@U03NZ7Z52S2> but fyi, these are in disabled state so we should not have any issues even if we don't delete them""}, {""user"": ""bganganna"", ""timestamp"": ""1689577913.791729"", ""content"": ""ok <@U02SF36PVKL> then we can delete the core-data entries ?""}, {""user"": ""mnirmal"", ""timestamp"": ""1689577929.359809"", ""content"": ""yeah we can""}, {""user"": ""aganivada"", ""timestamp"": ""1689584631.654139"", ""content"": ""<@U03NZ7Z52S2> PFA results for publisher, product, product_instance and category""}, {""user"": ""aganivada"", ""timestamp"": ""1689588592.872559"", ""content"": ""Updated counts\n\n```select count(*) from publisher where name like 'publisher-%';```\n2820\n\n> select count(*) from svc_coredata_default_axm.product_instance where name like 'product-%';\n1966\n\n> ```select count(*) from product where name like 'product-%';```\n> \n2161\n\nproduct_category = no changes""}, {""user"": ""aganivada"", ""timestamp"": ""1689589208.414109"", ""content"": ""We cleaned up the records of sys_test tenants currently mapped in sys_test_details.json, For the remaining records we may have to export the tenant_id's except \""0\"" from db to a text file and run a robot job to trigger deletion for each tenant.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689600939.221549"", ""content"": ""There are many records of tenants that are missing.. I guess that\u2019s where the bulk of the junk comes from""}, {""user"": ""aganivada"", ""timestamp"": ""1689655989.414919"", ""content"": ""Yes Rama, these are most likely tenants from pre tenant deletion coordination time. Bhavana already made changes to read TID's from a file for cleanup, once we test this in stage we can use the same on prod""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689657329.863479"", ""content"": ""Let's be very cautious""}, {""user"": ""aganivada"", ""timestamp"": ""1689657806.647069"", ""content"": ""we will take a db backup before running cleanup and run during weekend or next monday morning""}]" "1679580518.710029 ","[{""user"": ""aganivada"", ""timestamp"": ""1679580518.710029"", ""content"": ""<@U02GC8SE18V> do you have a sample payload for name resolution query for tenant table? I used typeName as \"":Tenant\"" but not sure what to pass in key :man-facepalming: cc: <@U03BPNY5AGM>""}, {""user"": ""rvaidya"", ""timestamp"": ""1679580645.241339"", ""content"": ""We are referring this enum : ""}, {""user"": ""rvaidya"", ""timestamp"": ""1679580650.872599"", ""content"": ""colon is not required now.""}, {""user"": ""rvaidya"", ""timestamp"": ""1679580677.669459"", ""content"": ""cc <@U034RLJA97X> <@U026PMDB1ND>""}, {""user"": ""aganivada"", ""timestamp"": ""1679580948.712509"", ""content"": ""<@U02GC8SE18V> what do we pass for the payload? I tried following but it doesnt work\n\n> {\n> \""input\"": [\n> {\n> \""typeName\"": \""Tenant\"",\n> \""key\"": {\n> \""id\"": 11256\n> }\n> }\n> ]\n> }""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679580988.185529"", ""content"": ""@anil, where are you logging this from?""}, {""user"": ""aganivada"", ""timestamp"": ""1679581051.518629"", ""content"": ""this is not for audit-log <@U026PMDB1ND>,\n\n<@U03BPNY5AGM> wanted a light weight name resolver when we have an id""}, {""user"": ""aganivada"", ""timestamp"": ""1679581085.648829"", ""content"": ""instead of fetching entire tenant details by id we thought we could use the name resolver API""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679581144.177529"", ""content"": ""Okay. Just have key: 11256""}, {""user"": ""aganivada"", ""timestamp"": ""1679581185.605199"", ""content"": ""perfect that worked, thank you <@U026PMDB1ND>\n\n> ```curl -X 'POST' \\\n> '' \\\n> -H 'accept: application/json' \\\n> -H 'Authorization: Bearer eyJraWQ' \\\n> -H 'Content-Type: application/json' \\\n> -d '{\n> \""input\"": [\n> {\n> \""typeName\"": \""Tenant\"",\n> \""key\"": 11256\n> \n> }\n> ]\n> }'```\n> ""}, {""user"": ""aganivada"", ""timestamp"": ""1679581209.958129"", ""content"": ""cc: <@U03BPNY5AGM>""}, {""user"": ""pkarthikeyan"", ""timestamp"": ""1679581242.397369"", ""content"": ""Thanks <@U02BV2DGUKC> <@U026PMDB1ND> <@U02GC8SE18V>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679581265.448489"", ""content"": ""key is an object because it can be long or string.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679581362.954339"", ""content"": ""Could probably have made it s string and converted internally ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679581558.953489"", ""content"": ""I suppose we\u2019d use use the Java classes generated by OpenAPI""}, {""user"": ""aganivada"", ""timestamp"": ""1679581669.514379"", ""content"": ""Yup that is the intent I guess, <@U03BPNY5AGM> can you confirm?""}, {""user"": ""pkarthikeyan"", ""timestamp"": ""1679581703.444759"", ""content"": ""Yes <@U026PMDB1ND>""}, {""user"": ""rvaidya"", ""timestamp"": ""1679582854.153549"", ""content"": ""<@U026PMDB1ND> can we use these name resolver api from apps ?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679598952.520679"", ""content"": ""Yes, <@U02GC8SE18V>. There's no reason to limit the this to the lambda.""}]" "1679065372.682699 ","[{""user"": ""ppant"", ""timestamp"": ""1679065372.682699"", ""content"": "" While debugging the base infra and coordination lambda changes, I came across these two API calls. These are the calls made to non existing APIs of ingestion and score-provider service (also at imprecise endpoints with `-service` appended at the end) which is throwing 404 but tenant deletion lambda is treating them as if the tenant didn\u2019t exist so the step functions didn\u2019t error out. Shall I add a check in the lambda to avoid these calls? Found this when the lambda got triggered with correct endpoints but these APIs started to throw 401\n\nLogs for 404 case \n\n401 - ""}, {""user"": ""askumar"", ""timestamp"": ""1679065705.662219"", ""content"": ""Possibly 404 is expected if the called service is not implementing the tenant deletion API.\nIn that case the tenant deletion lambda should not also throw error because that way it also knows that called service is yet to implement the tenant deletion API.""}, {""user"": ""ppant"", ""timestamp"": ""1679065898.392429"", ""content"": ""<@U03KLHDKL1H> Currently the endpoints of the services are wrong as they shouldn\u2019t have `-service`\n```ingestion_service: ServiceDefn = ServiceDefn(\n \""ingestion\"",\n \""INGESTION_SERVICE_HOST\"",\n \""INGESTION_SERVICE_PORT\"",\n \""INGESTION_SERVICE_SCHEME\"",\n)\n\nscore_provider_service: ServiceDefn = ServiceDefn(\n \""score-provider\"",\n \""SCORE_PROVIDER_SERVICE_HOST\"",\n \""SCORE_PROVIDER_SERVICE_PORT\"",\n \""SCORE_PROVIDER_SERVICE_SCHEME\"",\n)```\nThe issue comes when you call these services at their correct endpoints. In that case 401 is thrown which stops the step functions and the lambdas fails""}, {""user"": ""askumar"", ""timestamp"": ""1679066089.270439"", ""content"": ""Ohh okay...got it !!""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679066551.533799"", ""content"": ""<@U0431DZTPJM> - thanks for locating this. The fix would be in the statemachine json. I guess we didn\u2019t notice that because at the time we built the state machine not all services had integrated. These services may not have integrated yet and perhaps we can drop ingestion altogether from this list.""}, {""user"": ""ppant"", ""timestamp"": ""1679067061.283089"", ""content"": ""Sure <@U026PMDB1ND>, I will make the change""}]" "1683197749.854329 ","[{""user"": ""aganivada"", ""timestamp"": ""1683197749.854329"", ""content"": ""~ just got an alert that we are observing spike in logging can someone take a look? I am afk now~""}, {""user"": ""askumar"", ""timestamp"": ""1683197868.936549"", ""content"": ""Looking <@U02BV2DGUKC>""}, {""user"": ""askumar"", ""timestamp"": ""1683198429.449249"", ""content"": ""In production observing WARN logs:\n\n\nGetRecords request throttled for [ShardConsumer{shardOffset=KinesisShardOffset{iteratorType=LATEST, sequenceNumber='null', timestamp=null, stream='production_us-west-2_axm-config-events', shard='shardId-000000000002', reset=false}, state=CONSUME}] with the reason: Rate exceeded for Shard - 643306803378/production_us-west-2_axm-config-events/shardId-000000000002\t -\n\n\n""}, {""user"": ""aganivada"", ""timestamp"": ""1683198822.924349"", ""content"": ""<@U03KLHDKL1H> this is retryable exception and usually service recovers on retrying. Maybe the stream is getting overloaded will take a look. Wondering if we turned on debug log or some exceptions in int and stage causing this excessive logging.""}, {""user"": ""askumar"", ""timestamp"": ""1683199179.807439"", ""content"": ""Few services appears to be having Debug level logs in INT\n\n""}, {""user"": ""aganivada"", ""timestamp"": ""1683199385.073609"", ""content"": ""looks like a false alarm I had set threshold across env's to myself long time ago while we were testing something which seems to be triggering once in a while. One good thing is we figured the kinesis throttling issue, I will add an alert on this and check if we need to increase resources on kinesis front or reduce the polling interval""}, {""user"": ""askumar"", ""timestamp"": ""1683199441.795029"", ""content"": ""cool...thanks""}, {""user"": ""aganivada"", ""timestamp"": ""1683199471.695419"", ""content"": ""> Few services appears to be having Debug level logs in INT\nsize looks ok for 1 day, not that bad""}]" "1689140516.958359 ","[{""user"": ""aganivada"", ""timestamp"": ""1689140516.958359"", ""content"": ""<@U04JT69T00K> we might also have to check if AWS allows one proxy can serve multiple rds (for prod), based on pricing logic it doesn't seem possible?""}, {""user"": ""pjha"", ""timestamp"": ""1689140802.775659"", ""content"": ""yes, here it's mentioned each proxy can have single target.""}]" "1689004394.218519 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1689004394.218519"", ""content"": ""<@U02BV2DGUKC> - When we send kinesis stream to s3 , \u201cNewline delimiter\u201d needs to be enabled . If it is enabled , we can easily load in athena and do all sql filters /group by - It will be helpful to find duplicates or any other issues - Could you add a backlog in platform for making this change in CDK ?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689004569.701939"", ""content"": ""I have created a table for inbound-pulses in athena for the same but bcoz of the missing delimiter , it is showing only first recird in a batch\n```CREATE EXTERNAL TABLE inboundpulses (\n\tpayload struct < \n tenantId: bigint,\n productId: bigint,\n productInstanceId: bigint,\n pvtId: bigint,\n numRecipients: bigint,\n responded: boolean,\n userId: bigint,\n personaId: bigint,\n pulseSurveyUserInstanceId: bigint,\n personaWeight: decimal,\n vendorId: bigint,\n pulseId: bigint,\n pulseTemplateId: bigint,\n pulseScheduleId: bigint,\n pulseEndBroadcastTime: string,\n pulseTemplateTypeId: bigint,\n pulseInstanceId: bigint,\n pulseInstanceStartTime: string,\n pulseInstanceTtl: string,\n pulseInstanceResponseId: bigint,\n pulseInstanceResponseTime: string \n\t>\n)\n\n\nROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'\nLOCATION '';\nmsck repair table inboundpulses;\nselect *\nfrom inboundpulses ;```""}, {""user"": ""aganivada"", ""timestamp"": ""1689005535.811379"", ""content"": ""<@U0336QZAF98> added for this. for now can we manually enable this flag? if it is a blocker we can start looking into it.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689005580.390279"", ""content"": ""Let me know if I can make change in prod""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689005605.311069"", ""content"": ""Or if Prashant can make it for data related buckets it will be helpful""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689005678.148889"", ""content"": ""<@U0336QZAF98> - what happens if we make it apply to all buckets instead of just data related buckets?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689005798.909979"", ""content"": ""This can be enabled for all kinesis streams with dynamic partitioning enabled - for data buckets it is there - for some we have not enabled dynamic partitioning""}, {""user"": ""aganivada"", ""timestamp"": ""1689005952.685759"", ""content"": ""> Let me know if I can make change in prod\n<@U0336QZAF98> ticket was added to configure it via cdk we might also have to test before deploying cdk change to prod. Meanwhile if this is blocking we can probably go ahead and update manually for now""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689006434.327179"", ""content"": ""Here is the way to do this in CDK <@U04JT69T00K>\n\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689006468.239119"", ""content"": ""``` {\n type: 'AppendDelimiterToRecord',\n parameters: [\n {\n parameterName: 'Delimiter',\n parameterValue: '\\\\n',\n },\n ],\n },```\nin the processors section""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689006710.684549"", ""content"": ""We have a kinesis_stack and kinesis_delivery_stream_stack.. Looks like we could have refactored to eliminate some duplication but here we are...""}]" "1677848844.306539 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1677848844.306539"", ""content"": ""<@U040RCBPBEC> <@U02Q02G09QX> - Please review topics generated for last pass data - it will be great if we can go over quick sycn up call on the same""}, {""user"": ""svummidi"", ""timestamp"": ""1677857020.899329"", ""content"": ""I checked quickly, looking good to me.\nIn your last test results, noticed, topics very lengthy - But I don\u2019t find them in this data.\nCan you get topics of max length and topics of smallest size - This is one way to validate the quality of the topics.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1677957995.248729"", ""content"": ""<@U040RCBPBEC> - i modified the prompt and reran against a sample of 250 records .\n\n\nBelow is modified prompt - where keywords are ranked plus introduced emotion and product aspect - let me know your thoughts\n``` Lastpass is a password manager that helps you store, manage, and secure all your passwords in one place. \n I have given a survey question and answer about LastPass product . \n Based on the answer in the context of question, \n 1.Detect overall sentiment - positive , negative or neutral and sentiment score from 0-1 \n 2.If overall sentiment is positive , extract 1-10 positive keywords sorted by sentiment score in descending order. \n 3.If overall sentiment is negative , extract 1-10 negative keywords sorted by sentiment score in descending order. \n 4.Extract neutral keywords related to product feature \n 5.Based on keywords extracted from the answer find matching emotions - satisfied , unsatisfied , frustrated , excited ,thrilled, bored, thankful , regretful , burdened, relieved , amazed \n 6.Based on keywords extracted from the answer find matching product aspects - performance ,security,cost, product features, usability , customer support , documentation\n 7. Below is sample output\n { \""overallSentiment\"":\""Neutral\"", \""sentimentScore\"":0.8, \""matchedEmotions\"":[\""a\"",\""b\""],\""matchedAspects\"":[\""a\"",\""b\""], \""positiveKeywords\"":[\""a\"",\""b\""],\""negativeKeywords\"":[\""a\"",\""b\""], \""neutralKeywords\"":[\""a\"",\""b\""] }```""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678011127.476139"", ""content"": ""With the updated prompt - ran our retro comments. ""}, {""user"": ""akalyta"", ""timestamp"": ""1678124765.723759"", ""content"": ""<@U0336QZAF98> <@U040RCBPBEC> interesting article on 3.5turbo vs davinci ""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678124906.020209"", ""content"": ""yes <@U02Q02G09QX> - i have switched to gpt-3.5 for poc - it looks good - The one of drawbacks - you cannot train your own model on gpt 3.5 yet - but apart from that it is cheaper , faster""}, {""user"": ""akalyta"", ""timestamp"": ""1678125006.215359"", ""content"": ""<@U0336QZAF98> yes! I\u2019ve explored your document and it looks very intriguing. I am curious if we can use it for summarization too given that it tends to perform better on 0-shot problems""}, {""user"": ""svummidi"", ""timestamp"": ""1678218476.344119"", ""content"": ""<@U033PPLNFRU> - In the below link, \u201cturbo_topics\u201d column, last entries represent the emotion.\n""}]" "1692763234.256229 ","[{""user"": ""aganivada"", ""timestamp"": ""1692763234.256229"", ""content"": ""<@U03KLHDKL1H> are we good with system_user masking changes in invitation email ? can we make this change as a HF? cc: <@U033PPLNFRU>""}, {""user"": ""askumar"", ""timestamp"": ""1692764329.715989"", ""content"": ""<@U02BV2DGUKC> our template changes were done.\n\nFor tenancy I have made code changes, will get them reviewed once with you before MR and make them as part of hotfix.""}, {""user"": ""askumar"", ""timestamp"": ""1692801724.560529"", ""content"": ""<@U033PPLNFRU> <@U02BV2DGUKC>\nThe changes of new template are deployed to STAGE.\nTested by adding user to engagement via landing page.\n\nThanks""}, {""user"": ""askumar"", ""timestamp"": ""1692805995.313029"", ""content"": "" <@U03NZ7Z52S2>\nCould you also please verify it in stage, so that we can move this to production?""}, {""user"": ""bot_message"", ""timestamp"": ""1692806212.211269"", ""content"": ""@Anu created a Task TEST-1095 New email template validation""}, {""user"": ""bganganna"", ""timestamp"": ""1693398230.908349"", ""content"": ""<@U03KLHDKL1H> <@U02BV2DGUKC> I m still seeing the system_user name in the email if i add from teams page""}, {""user"": ""bganganna"", ""timestamp"": ""1693399100.231069"", ""content"": ""<@U02BV2DGUKC> As system user r we not suppose to invite vendor? It fails with message: Unable to find user. (64ef313b3e209bc7c4a8b035f9566cd5)""}, {""user"": ""aganivada"", ""timestamp"": ""1693400076.384789"", ""content"": ""<@U03NZ7Z52S2> ideally system user is not supposed to invite vendor but if we have to absolutely do it then we need to add system user as a piu for the tenant and then invite""}, {""user"": ""aganivada"", ""timestamp"": ""1693400144.405159"", ""content"": ""Also regarding the email it is sent from invite user flow from admins page, The flow which Ashwani fixed was inviting EO flow""}, {""user"": ""aganivada"", ""timestamp"": ""1693400213.678189"", ""content"": ""Can you try sending invite as a system user as engagement owner or initiative owner etc""}, {""user"": ""bganganna"", ""timestamp"": ""1693400288.190989"", ""content"": ""ok let me try""}, {""user"": ""bganganna"", ""timestamp"": ""1693400725.851829"", ""content"": ""<@U02BV2DGUKC> I dont see in the email, but visible after accepting the invite""}, {""user"": ""aganivada"", ""timestamp"": ""1693400793.493969"", ""content"": ""hmmm ... let me check in auth0 <@U03NZ7Z52S2>""}, {""user"": ""aganivada"", ""timestamp"": ""1693401611.081459"", ""content"": ""<@U03NZ7Z52S2> can you try now?""}, {""user"": ""aganivada"", ""timestamp"": ""1693401905.269979"", ""content"": ""<@U033PPLNFRU> for the text in auth0 when user accepts invitation we have to entirely remove inviters email if we want to hide system_user.\n\ndefault message text:\n```Log in to accept ${inviterName}'s invitation to join ${companyName} on ${clientName}.```\nUpdated message text:\n```Log in to accept invitation to join ${companyName} on ${clientName}.```\nPlease let us know if it looks ok otherwise as Bhavana mentioned though we are hiding system_user from email text system user will still show up in auth0 prompt ""}, {""user"": ""bganganna"", ""timestamp"": ""1693415458.071629"", ""content"": ""> can you try now?\n<@U02BV2DGUKC> i still see the system_user name on sign up page""}, {""user"": ""aganivada"", ""timestamp"": ""1693415588.577089"", ""content"": ""ok, <@U03NZ7Z52S2> can yo share the full message that we see?""}, {""user"": ""bganganna"", ""timestamp"": ""1693415707.738339"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1693417004.711939"", ""content"": ""<@U03NZ7Z52S2> can we check now? updated\n\n```Sign Up to accept ${inviterName}'s invitation to join ${companyName} on ${clientName}.```\nto\n```Sign Up to accept invitation to join ${companyName} on ${clientName}.```\nprevious change for cases where existing user receives an invitation""}, {""user"": ""bganganna"", ""timestamp"": ""1693417237.933009"", ""content"": ""<@U02BV2DGUKC> not showing the username now""}, {""user"": ""aganivada"", ""timestamp"": ""1693417479.188879"", ""content"": ""there is no way to hide specifically for system_user so we have to hide entire inviter field""}, {""user"": ""aganivada"", ""timestamp"": ""1693417523.548999"", ""content"": ""this is a setting in auth0, we wont need a code change for this <@U03NZ7Z52S2>, lets wait for confirmation from <@U033PPLNFRU> then we can make the same change in production auth0 account""}, {""user"": ""anair"", ""timestamp"": ""1694032707.589049"", ""content"": ""<@U02BV2DGUKC> <@U03NZ7Z52S2>\nWhere are we fetching {inviterName} from? Can we rename system_user in that DB to \""your org\""? This will handle the issue without completing removing inviterName field which is important context""}, {""user"": ""aganivada"", ""timestamp"": ""1694057206.670349"", ""content"": ""<@U033PPLNFRU> {inviterName} is coming from auth0, we don't store usernames in our DB. The screenshot that Bhavana shared is an auth0 page where they lookup for message text and replace content based on user activity. They allow us to customize the content. so we can replace {inviterName} with \""your org\"".""}]" "1685545131.831239 ","[{""user"": ""aganivada"", ""timestamp"": ""1685545131.831239"", ""content"": "" please review we can use this during our planning meeting""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685579777.189809"", ""content"": ""<@U02BV2DGUKC> - can't we use Swanly instead of/in addition to this?""}, {""user"": ""aganivada"", ""timestamp"": ""1685580634.092369"", ""content"": ""Yes <@U026PMDB1ND> will move this to swanly and track from there, I found confluence a bit easy for planning than swanly. Tracking execution wise Swanly is definitely very helpful.""}]" "1686720987.555169 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1686720987.555169"", ""content"": ""<@U03KLHDKL1H> - can you reshare the Snowflake creds here and some sample queries? Roughly how much time is taken to execute a query against the metrics we've stored (how many records are we running this query against?)\n\ncc <@U02D4DUKDQC>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686721223.010609"", ""content"": ""<@U02D4DUKDQC> - for storing the customer metrics we could potentially use Snowflake. When I last looked at it, driving a Web UI was a bit challenging against smallish amount of data. Where it shines is when there's a ton of data. The cost equation is dramatically better (compared to ES, for instance) when the data is large and most of it is dormant.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686721559.530449"", ""content"": ""<@U040RCBPBEC> - for the kind of metric data that the PM is talking about, we should be able to use Druid, right?\n\nWhat are the biggest risks of using Druid for this use case, in your opinion?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686721616.278389"", ""content"": ""<@U02GC8SE18V> - when you build value metrics in SHN, did you use Crate for storage? Did we consider using Druid?""}, {""user"": ""askumar"", ""timestamp"": ""1686722327.356389"", ""content"": ""<@U026PMDB1ND>\nPlease use the user name: AXIAMATIC\nPassword in 1password :\nhere is the link \n\nDashboard link.\n\n\n~For query time : 1.1 K records in PulseInstance table~\n~Average time around 1-2 seconds~\n""}, {""user"": ""askumar"", ""timestamp"": ""1686722495.480949"", ""content"": ""Correction : These queries are from Eventlog table around 85000 records\nTime 1-2 seconds""}, {""user"": ""askumar"", ""timestamp"": ""1686723189.541729"", ""content"": ""Sample queries :\n---\nWITH subquery AS (\n SELECT event:type as type, event:env as env\n FROM eventlog\n where type in ('pulse-survey-sent-event','pulse-survey-button-clicked-event','pulse-survey-submitted-event')\n and event:env = 'int'\n )\nSELECT count(*),subquery.type\nFROM subquery\ngroup by subquery.type\n---\n\n---\n\nWITH subquery AS (\n SELECT event:payload:pulseInstanceId as instanceId, event:payload:productInstanceUserId as userId, event:type as type, event:payload:tenantId as tenantId, event:env as env\n FROM eventlog\n where type in ('pulse-survey-sent-event','pulse-survey-button-clicked-event','pulse-survey-submitted-event')\n and event:env = 'int'\n )\nSELECT count(*), subquery.instanceId, subquery.userId, subquery.type, subquery.tenantId, subquery.env\nFROM subquery\ngroup by subquery.instanceId,subquery.userId,subquery.type,subquery.tenantId, subquery.type,subquery.env\n\n------""}, {""user"": ""rvaidya"", ""timestamp"": ""1686723816.656949"", ""content"": ""when you build value metrics in SHN, did you use Crate for storage? Did we consider using Druid?\n\n<@U026PMDB1ND> we used Victoria db, ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686725211.714819"", ""content"": ""<@U02GC8SE18V> - did we consider Druid and not find it suitable?""}, {""user"": ""rvaidya"", ""timestamp"": ""1686725312.415659"", ""content"": ""Victoria worked smoothly with Prometheus <@U026PMDB1ND>, we didn't consider druid as we were looking more from Prometheus compatibility""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686725821.075589"", ""content"": ""<@U02D4DUKDQC> 1-2 seconds is not bad. We need to get some rough numbers (in conjunction with PM) regarding the amount of data we'd probably see in the next 6 months to 1 year, push some cooked up data and do some quick prototyping (essentially look at query times of the most likely queries).\n\nIn my previous gig, we pointed SF at a S3 bucket and programmatically generated some data, pushed it to S3 so that SF would pick it up.""}, {""user"": ""svummidi"", ""timestamp"": ""1686754555.940819"", ""content"": ""<@U026PMDB1ND> Druid is good for scale and performance but it\u2019s major drawback is immutability.\nWhat is 1-2 seconds with SF?\n> <@U02D4DUKDQC> 1-2 seconds is not bad. We need to get some rough numbers (in conjunction with PM)""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686754635.845059"", ""content"": ""<@U040RCBPBEC> Ashwani shared a few sample queries with Snowflake in thus thread where the time taken to execute the query was 1-2 seconds ""}, {""user"": ""svummidi"", ""timestamp"": ""1686754884.481779"", ""content"": ""<@U026PMDB1ND> - Probably it is for initial queries but if it is consistent, it can be a problem for our usage pattern. We need to aim for sub-second response time otherwise it adds to visible delay in UI due to limited concurrent connections and need for executing multiple queries to render a page.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686755445.070499"", ""content"": ""Since SF is an OLAP system and not a OLTP system and because of the way they store data, we cannot expect significantly better times. If we are going to get a lot of metrics data and need major flexibility SF is a good candidate, but timings are going to be in this range. We could scale things up on the SF side to handle more connections and get similar times but in general, we get progressively slower times depending on how much data we need to pull out to compute.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686755824.907869"", ""content"": ""They store data in S3 in segments and keep a master index in a different DB. Then they fetch the segments from S3 by computing what is in the master DB and execute the query. The writes to a segment managed by versioning so the readers don't need to worry about any sort of contention. There's some smart caching if the queries hit the same segments over and over.\n\nThey are very cost-effective when the data needed to process is large because they don't need to keep a lot of data in memory or need fast SSD to store lots of data.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686755982.156509"", ""content"": ""I think the key for us is to store the metadata along with the metric so that we can flexibly reprocess the data.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686756267.164719"", ""content"": ""<@U02D4DUKDQC> - we may need to do this POC without actually trying out all the options :slightly_smiling_face:""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1686756341.683939"", ""content"": ""<@U026PMDB1ND> If we are trying to keep things similar to value score does it make sense to consider druid as well along with snowflake""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686756424.601799"", ""content"": ""<@U02D4DUKDQC> - the challenge with Druid as Satya clarified above is the flexibility aspect. We need to explore that some more. The reason why we got Druid in the first place was to be able to support this use case :slightly_smiling_face:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686756501.082869"", ""content"": ""Another option we can think of is dumping data into SF, running a pipeline of sorts on top of that data to generate 2nd order data that we'd store in Druid and drive UI off Druid.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686756688.509919"", ""content"": ""This is how some people use SF when they need to drive UI - store the raw data in SF and store the synthesized data in a faster store (RDBMS etc.) Basically to bridge OLAP world which can deal with arbitrarily large amounts of data with the needs of a UI that can be faceted etc.""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1686756819.178879"", ""content"": ""That probably will be needed if lows and highs of metrics can be edited which impacts normalisation""}]" "1684814536.276599 ","[{""user"": ""aganivada"", ""timestamp"": ""1684814536.276599"", ""content"": ""<@U0431DZTPJM> please take a look at there seems to be some issue with email template \""collab-web-survey-v1\"".\n\n<@U03NZ7Z52S2> this is not related to the VPN changes we made since there is no issue with connectivity but will take a look if we can find the root cause from notification service perspective do you recollect when was this last tested?""}, {""user"": ""bganganna"", ""timestamp"": ""1684816140.399999"", ""content"": ""Mostly in the last release <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1684848477.646879"", ""content"": ""<@U0431DZTPJM> any luck with this?""}, {""user"": ""ppant"", ""timestamp"": ""1684849631.380699"", ""content"": ""I looked at it in the morning. The template seemed correct, tested it in but since then got occupied in the axm api client thing.\n\n<@U03NZ7Z52S2> Can you share curl request to run the API in int which triggers sending this email?""}, {""user"": ""aganivada"", ""timestamp"": ""1684851627.309219"", ""content"": ""<@U0431DZTPJM> could it be that stage has a different template?""}, {""user"": ""ppant"", ""timestamp"": ""1684851782.622989"", ""content"": ""Ummm, so in stage email-templates was deployed and changes in web-survey template were made yesterday""}, {""user"": ""aganivada"", ""timestamp"": ""1684851869.286719"", ""content"": ""Ok so int has different template? We did upload templates to stage last weekend as part of deployment""}, {""user"": ""aganivada"", ""timestamp"": ""1684851922.134069"", ""content"": ""Can we check in db of int on when template was updated""}, {""user"": ""ppant"", ""timestamp"": ""1684852084.937539"", ""content"": ""int has the latest template""}, {""user"": ""ppant"", ""timestamp"": ""1684852120.658469"", ""content"": ""And I guess stage must be having older version so there might be some handle bar variables that are mismatching/absent""}, {""user"": ""aganivada"", ""timestamp"": ""1684852227.556179"", ""content"": ""Yeah makes sense\n\n<@U02SF36PVKL> do you know if we need to update websurvey template in stage? We are getting handle bar errors in stage <@U0431DZTPJM> checked that the latest one in int seems to be working fine""}, {""user"": ""ppant"", ""timestamp"": ""1684852262.004749"", ""content"": ""There are some new placeholder variables in this commit 3 days ago ""}, {""user"": ""mnirmal"", ""timestamp"": ""1684852308.476739"", ""content"": ""<@U02BV2DGUKC> yes we need to, I was planning on pinging you tomorrow morning. Can we deploy 0.0.18.1 SC on stage and run ./run-local.sh with stage env?""}, {""user"": ""mnirmal"", ""timestamp"": ""1684852364.721679"", ""content"": ""Sorry <@U03NZ7Z52S2> <@U02BV2DGUKC> missed informing about the delay.""}, {""user"": ""aganivada"", ""timestamp"": ""1684852389.572199"", ""content"": ""Sure np <@U02SF36PVKL> should I deploy now?""}, {""user"": ""mnirmal"", ""timestamp"": ""1684852402.144069"", ""content"": ""yes <@U02BV2DGUKC> you can""}, {""user"": ""aganivada"", ""timestamp"": ""1684852410.011569"", ""content"": ""Ok""}, {""user"": ""aganivada"", ""timestamp"": ""1684854451.473709"", ""content"": ""<@U02SF36PVKL> assigned to you cc: <@U0431DZTPJM>""}]" "1686556314.545279 ","[{""user"": ""askumar"", ""timestamp"": ""1686556314.545279"", ""content"": ""<@U026PMDB1ND> <@U02BV2DGUKC>\nIn INT we are seeing a high CPU utilisation of Aurora clusters, although we have 5 services right now migrated.\nWe have kept vCPU as Max 2 Min 0.5.\n\nNeed help on following:\nWhat should be the max vCPU that we should configure in INT ?\nOr should we consider option of multiple cluster, since number of connections can cross 300 in INT with services and people also trying to connect.\n\ncc ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686673597.249239"", ""content"": ""<@U03KLHDKL1H> - can you share the link to the graph?""}, {""user"": ""askumar"", ""timestamp"": ""1686673655.365429"", ""content"": """"}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686673673.376159"", ""content"": ""My initial take is:\n\na) We should not go with multiple clusters\nb) We should just increase the min and max as needed""}, {""user"": ""askumar"", ""timestamp"": ""1686673761.664099"", ""content"": ""I reduced it to 1 second average, was seeing 100% rates.\nRight now we have 2 vCPU, should we increase it to 8 (upper limit)?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686673826.940519"", ""content"": ""We don't need 8, I think. Do you remember the sizes of RDS we had?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686673858.595069"", ""content"": ""nvm db.t3.medium and db.t3.small""}, {""user"": ""askumar"", ""timestamp"": ""1686673871.105169"", ""content"": ""Yeah""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686673986.591379"", ""content"": ""We 2 small and 1 medium, which is 6 cores and 8G.\n\n\n\""Each ACU is a combination of approximately 2 gibibytes (GiB) of memory, corresponding CPU...\""""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686674005.340619"", ""content"": ""So 4 ACU = 8G and whatever is the CPU that goes with it""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686674076.090619"", ""content"": ""We could go to 0.5 min, 3 max ACU and see how it goes.""}, {""user"": ""askumar"", ""timestamp"": ""1686674115.056759"", ""content"": ""Sure <@U026PMDB1ND>\nWould work with <@U04JT69T00K> to increase it.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686674120.060839"", ""content"": ""If that's is not good enough, 1 min and 3 max.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686674168.527619"", ""content"": ""Btw, we just have one service connected so far? Also, there's a proxy concept that is supported which reduces the DB connection requirement. We should look at that as well, specifically in INT""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686674183.826279"", ""content"": ""Please do check with Prashant if he can take care of this""}, {""user"": ""askumar"", ""timestamp"": ""1686674216.168749"", ""content"": ""Actually there are 5 services connected (Disc, App, Core).""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686674261.947029"", ""content"": ""Ok. The rubber meets the road when Collab is moved over :slightly_smiling_face:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686674281.812079"", ""content"": ""I don't think apps & disc hit the DB as hard as core.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686674308.903989"", ""content"": ""Maybe reports on the apps side does, but it is not a constant load.""}, {""user"": ""askumar"", ""timestamp"": ""1686674342.528609"", ""content"": ""Yeah :slightly_smiling_face:, About 160 connections in RDS commons, where Collab is with others.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686753760.096199"", ""content"": ""<@U04JT69T00K> can you look into the possibility of using a proxy in front of RDS""}, {""user"": ""pjha"", ""timestamp"": ""1686753871.272749"", ""content"": ""Yes, sure <@U026PMDB1ND> ""}]" "1683784308.556359 ","[{""user"": ""gshenoy"", ""timestamp"": ""1683784308.556359"", ""content"": ""<@U02BV2DGUKC> Is there any change to tenancy being tested ? We are seeing 403 on tenancy/api/v1/tenants/33606/users/permission API""}, {""user"": ""aganivada"", ""timestamp"": ""1683784713.025699"", ""content"": ""<@U028EDANJM9> are you trying through VPN?""}, {""user"": ""gshenoy"", ""timestamp"": ""1683784814.043459"", ""content"": ""No <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1683784971.998859"", ""content"": ""ok we enabled VPN on tenancy, vault and notification service so if we are attempting from local we have to sign in to VPN.""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1683800344.324249"", ""content"": ""+ <@U03DLS1FN3D>""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1683800490.066509"", ""content"": ""<@U02BV2DGUKC> This API is failing when generating reports and requests are made from Lambda vis puppeteer. Due to this we are not able to generate reports in int, Can you please help with this. cc <@U02SCRTM2M7> <@U03DLS1FN3D>""}, {""user"": ""aganivada"", ""timestamp"": ""1683800570.397359"", ""content"": ""<@U02HCMTQU3W> lambda's should be talking over private dns let me check with <@U0431DZTPJM>""}, {""user"": ""aganivada"", ""timestamp"": ""1683800742.629179"", ""content"": ""<@U02SCRTM2M7> can you pass me the token we send to dashboard-app in int? ""}, {""user"": ""aganivada"", ""timestamp"": ""1683800792.396339"", ""content"": ""we didn't update authorizer attached to dashboard app so not sure why we are getting unauthorized access""}, {""user"": ""ppant"", ""timestamp"": ""1683801024.739269"", ""content"": ""<@U02BV2DGUKC> The reporting lambda is calling all the 3 services integrated in it at their private endpoints, checked this with <@U02SCRTM2M7> today""}, {""user"": ""aganivada"", ""timestamp"": ""1683801090.922389"", ""content"": ""ok thank you <@U0431DZTPJM> , is Syed around can we get on a call? want to check the token we are passing to dashboard app from lambda""}, {""user"": ""ppant"", ""timestamp"": ""1683801125.889909"", ""content"": ""We checked the JWT token, it looked fine. Will post it here once he is back""}, {""user"": ""aganivada"", ""timestamp"": ""1683801129.477109"", ""content"": ""probably they are seeing some issue while connecting to vault""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1683801153.550679"", ""content"": ""<@U02BV2DGUKC> let me know if i need to check anything on vault service""}]" "1673333151.941449 ","[{""user"": ""rvaidya"", ""timestamp"": ""1673333151.941449"", ""content"": ""<@U026PMDB1ND> <@U033PPLNFRU> Can you pls point me to the audit log event wiki ?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673333215.017199"", ""content"": ""Here it is <@U02GC8SE18V> : ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673333229.509449"", ""content"": ""Can you access it?""}, {""user"": ""rvaidya"", ""timestamp"": ""1673333262.966179"", ""content"": ""Yes, Thank You !""}, {""user"": ""svummidi"", ""timestamp"": ""1673379610.142669"", ""content"": ""I just want to share my views on IDs and names\u2026\n\nFor obvious reasons ID alone is not sufficient in audit logs. We need normalized details up to the level to explain the audit event.\n\nCapturing ID and then resolving it with lambda comes with more work, more moving items in our our infra. The biggest concern is - ID resolution is not guaranteed.\n\nWhen we complete a business transaction from user point of view, we need to make sure we have sufficient details to create audit log to describe the user action. Even if we need to make additional lookup before executing operation, still it is worth.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673384350.771089"", ""content"": ""Makes sense. In the create case (single or bulk), apps should have the name available (mostly) in the request context. For delete cases, we should definitely get the name and then delete, because anything that looks up after deletion is not likely to have the name available.\n\nOnly in update (single or more importantly, bulk) could there be some advantage in recording an Id and resolving it later. \n\nI think we should look at backend services and implement simple APIs that just return names given Ids. We could drive these off something like Redis later on if hitting RDB becomes costly (not an immediate need). Another thing we could do opportunistically is return the name of the object on update to avoid a round trip.\n\n\nOne challenge is if we should convert everything to a name in the context. Say if we invite a collaborator with a persona x to a pvt y, should we convert x and y to names at the time of populating the audit log event or should we resolve later. I worry that we\u2019d slow down everything by adding extra steps for this checkmark feature that most people may never even click into.\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673384814.422639"", ""content"": ""One option is to build a single name resolver api at the service level for many types\u2026\n\nInput could be\n```[\n {\n \""type\"": \""type name/enum...\"",\n \""ids\"": [.... ids ....]\n },\n {\n \""type\"": \""type name/enum...\"",\n \""ids\"": [.... ids ....]\n }\n .....\n]```\nOutput could be\n\n```[\n {\n \""type\"": \""type name/enum...\"",\n \""resolutions\"": [\n { \""id\"": ...., \""name\"": \""<resolved-name>\""},\n { \""id\"": ...., \""name\"": \""<resolved-name>\""},\n ] \n }\n .....\n]```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673390229.634239"", ""content"": ""<@U02GC8SE18V> can you please take the lead in terms what would be needed from the backend services to provide names\n\n(a) for the target object(s)\n(b) for related objects in the context\n\nfor C, U, D operations?\n\nWould be useful to understand how common is (b) in the first place. I am probably giving it too much weight based on the <collaborator, PVT, persona> example\n\nWe can consider (a) as a must have and (b) as good to have at the time of posting an audit log event.\n\nfyi <@U02D4DUKDQC> <@U028EDANJM9> <@U02BV2DGUKC>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673391748.942309"", ""content"": ""Btw. this is a WIP audit log search screen that <@U033PPLNFRU> shared:\n\nAryan: Do we expect to have a product context for every audit log?\n\nIt might be better to simplify the interface initially to just search for actions by one or all *users*, where the *type* of the object is one or all or many, and *action (create, update, delete, login...)* is one or many for a given *time frame*.\n\nWe shouldn't open up the interface to search for as many days as they want. We would incur cost for keeping this dead data around so maybe we don't want to offer a lengthy retention period to everyone.""}, {""user"": ""rvaidya"", ""timestamp"": ""1673407406.604759"", ""content"": ""<@U026PMDB1ND> I can schedule sometime on friday or early next week to start discussing backend dependencies. By then , i will get sometime to look at it once again as whole.""}, {""user"": ""anair"", ""timestamp"": ""1673407562.253869"", ""content"": ""<@U026PMDB1ND> is this data useful without the product context?\non the timeframe, will defer to whatever is your recommendation on data archival""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673407688.726839"", ""content"": ""<@U033PPLNFRU> the question is would every event have a product context""}, {""user"": ""anair"", ""timestamp"": ""1673407709.535239"", ""content"": ""<@U026PMDB1ND> no we would have global vs product specific actions""}, {""user"": ""anair"", ""timestamp"": ""1673407750.819319"", ""content"": ""<@U026PMDB1ND> I think the central question is whether we are building comprehensive audit log vs a stop gap that will appease EA. If latter then we can be very simple in the information we show""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673407887.462899"", ""content"": ""I think that it makes sense to build it to a decent level and be done with it (or with most of it) than to keep building it continuously.""}, {""user"": ""anair"", ""timestamp"": ""1673408659.190849"", ""content"": ""<@U026PMDB1ND> so if we define a minimum viable for audit log, would you agree with this\n\u2022 name instead of user ID\n\u2022 data stored for x days\n\u2022 single product vs all \n\u2022 *delete events -* all events captured\n\u2022 *login events -* all events captured\n\u2022 NO subject field\n\u2022 *create events -* some captured\n\u2022 *update events* - some captured""}, {""user"": ""anair"", ""timestamp"": ""1673408824.912199"", ""content"": ""<@U026PMDB1ND> adding a pulse template is global vs send pulse is product specific""}, {""user"": ""aganivada"", ""timestamp"": ""1673409492.215849"", ""content"": ""<@U033PPLNFRU> we are currently using this epic ""}, {""user"": ""anair"", ""timestamp"": ""1673409504.430589"", ""content"": ""Thanks <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1673410095.575439"", ""content"": ""<@U026PMDB1ND> just a thought, for delete cases would it make sense to flag a record as archived (similar to is_deleted) might be an overkill for audit log but looking at our pulse template config I think at some point there might be a request to recover data. If we look at our current objects, instances that have names are limited user actions, products, pulse templates, pulse instances etc., so if we can add add archived date column to each of these and update our filters we might be able to get the name even in deleted cases. We can have a cleanup job to permanently delete data that has been archived for more than x days, x in this case could be something greater than audit log purge limits.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673413393.097949"", ""content"": ""<@U02BV2DGUKC> - I don't think we should make a whole lot of deep changes for the sake of audit log which I suspect would be used extremely rarely.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673413914.799959"", ""content"": ""<@U033PPLNFRU> - please see inline\n> \u2022 name instead of user ID ====> today we are storing the user-email. We are not storing a user ID or name. Basically, we keep whatever the user logs in with. <@U02BV2DGUKC> if we have to capture the user name how would we do this?\n> \u2022 data stored for x days ===> Yes\n> \u2022 single product vs all. ====> `When the user is searching for login events would we just drop what they select in the product field or have \""None\"" in the product drop down?`\n> \u2022 *delete events -* all events captured ===> Whatever you called out in the Audit Log requirements page, yes.\n> \u2022 *login events -* all events captured ===> Yes. Note that we have some limitations since we need to translate Auth0 provided events and map them to our tenants. For example, if someone tries to login using a bunch of @lvsands.com e-mails (like brute force attempts, we can't do much with those events as of now). \n> \u2022 NO subject field ===> Fine\n> \u2022 *create events -* some captured ==> Whatever you called out in the Audit Log requirements page, yes., \n> \u2022 *update events* - some captured ==> Whatever you called out in the Audit Log requirements page, yes.\n> \n""}, {""user"": ""aganivada"", ""timestamp"": ""1673414315.106069"", ""content"": ""> today we are storing the user-email. We are not storing a user ID or name. Basically, we keep whatever the user logs in with. <@U02BV2DGUKC> if we have to capture the user name how would we do this?\n<@U026PMDB1ND> we discarded using user name and started using user email at all places, but with latest settings page we have an option to add/update user name. As part of the work (settings page) we can add an endpoint to get user info by email.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673417560.641799"", ""content"": ""Let's add that end point, Anil. If we can have that information available in the TenantUserContext in apps, we can store things in the audit event at the time of creation.""}, {""user"": ""anair"", ""timestamp"": ""1673457938.469979"", ""content"": ""<@U026PMDB1ND> we can decrease the scope of what is called out in the audit log page based on complexity and need""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673458012.798779"", ""content"": ""<@U033PPLNFRU>, <@U02GC8SE18V> is analyzing what it takes to support all the stated requirements. We can decide after she is done.""}, {""user"": ""anair"", ""timestamp"": ""1673458030.609279"", ""content"": ""<@U02BV2DGUKC> to reiterate <@U026PMDB1ND>s point audit log is def only going to be used very rarely. Either EA just needs it to check a box or when something is missing (delete events)""}, {""user"": ""rvaidya"", ""timestamp"": ""1673496856.410569"", ""content"": ""Somehow <@U026PMDB1ND> I am inclining on this : \nWe can discuss more tomorrow and have some discussion on any other best practise.""}, {""user"": ""aganivada"", ""timestamp"": ""1673497272.882779"", ""content"": ""<@U033PPLNFRU> <@U026PMDB1ND> do we absolutely need the user name or can we use email if user name is not available? the issue is most of the users we have today (and in future) will not have a user name by default since we dont mandate this during registration or when we invite a new user. Only cases where a user would have a name is when they go to settings page and update the name which is a manual action.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673498612.045489"", ""content"": ""<@U02BV2DGUKC> - if they provide it could it make it to the TenantUserContext? We can do\n_Name (e-mail)_ when both are available and just _E-mail_ if the name is not available.""}, {""user"": ""aganivada"", ""timestamp"": ""1673498803.505669"", ""content"": ""> if they provide it could it make it to the TenantUserContext?\nyes <@U026PMDB1ND>, that will be a very small customization in auth0 to pass user name and we might have to update code that builds TenantUserContext to read this field.""}, {""user"": ""rvaidya"", ""timestamp"": ""1673498841.609109"", ""content"": ""> we might have to update code that builds TenantUserContext to read this field.\nShould be doable. Let me which field to look for? <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1673498954.579539"", ""content"": ""sure <@U02GC8SE18V>, we can take it up from platform team to update apps as we started making small in roads to apps code :slightly_smiling_face: . will let you know if we run into any issues.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673498984.638069"", ""content"": ""Famous last words :stuck_out_tongue:""}, {""user"": ""rvaidya"", ""timestamp"": ""1673499094.121759"", ""content"": ""You ppl are \u201cPRO\u201ds across the tech stack \u2026and as always you(plat) underestimates what you can do in apps :smile:""}, {""user"": ""anair"", ""timestamp"": ""1673499163.413269"", ""content"": ""<@U02BV2DGUKC> if there is no name we can show email. Or in general, we can also only default to email only""}, {""user"": ""aganivada"", ""timestamp"": ""1673499307.109569"", ""content"": ""<@U02GC8SE18V> you should see me when I try to review apps code :man-facepalming::skin-tone-2: , may be getting a chance to add some code will help me understand code better.""}, {""user"": ""rvaidya"", ""timestamp"": ""1673600215.420369"", ""content"": ""Minutes from our discussion here :\n1. Do we log tenant deletions ? > Not required as it is something that we are doing internally. \n2. Do we log only user interactions? > And only webapp ? Yes\n3. Do we assume that collaborator will always be the actor for audit-log? > Mostly yes since we want to log only user interactions.\n4. Should we store the id-name resolution at the time of storing the audit-log? > This might add some time/cost to the API endpoints in Apps since we dont know how much the customer will use the feature. Also app still need to call backend services to resolve the ids.\n5. Should we store the id-name resolution at the time of fetching the adit-log to show in the UI ? > This can cause issues when the object has been deleted/edited. We will hv no way to track the audit log back.\n6. Should we use kind of event notification to enrich the audit-log by triggering the enrichment from apps and lets each service/consumer do its enrichments against the audit-log event id? > <@U0336QZAF98> to explain more about this approach from the flow perspective.\nI think the main question still remains :\n1. Does apps do it? As apps have the most context about the operation/request in general? \n2. Do we enrich using the lamdba ? \n3. Or a third party service/audit-service doing this? >> Concern here is that audit log service should not be responsible for any id resolution.\nAlso , few more questions :\n1. Do we log failed states as well for POST/PUT/DELETE? \n2. We need to think about way to log context in Async events. \nFeel free to add any point that was discussed and not listed above.\n\n<@U026PMDB1ND> <@U02BV2DGUKC> <@U028EDANJM9> <@U02D4DUKDQC> <@U03BPNY5AGM> <@U0336QZAF98> <@U040RCBPBEC> <@U02HCMTQU3W> <@U02HQ78V9A5>""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1673604897.193129"", ""content"": ""<@U026PMDB1ND> This was thought process i had in mind . As this will avoid redis cache lookups plus load on api service\n\nHighlevel Flow\n\n\n1. For every audit log event , audit log cli or lambda- will extract ids which need to be enriched\n2. publish a separate enrichment request event to separate sqs queue \n3. Each enrichment request event contains audit_log_id,entity_id.entity_type\n4. Below functionality can be wrapped as utility class audit_log_enricher in audit log cli and used in all microservices\n5. Every microservice can listen to queue (filtered only for event types they support)\n6. For every enrichment request event - microservice will find name of entity and publish enrichment event response to another sqs topic\n\taudit_log_id\n\tentity_id\n\tentity_type\n\tattributes :{\n\t \u201cname\u201d:\u201c\u201d\n\t}\n 7. From this topic , we can have a lambda which does update call (merge document) based on audit log id\n \n\n\nRaw Audit log:\n{\n\t\u201cid\u201d: \u201caudit-log-1\"" ,\n\t\u201cobjects\u201d: [\n\t\t{\n\t\t\t\u201cid\u201d:\u201c\u201d,\n\t\t\t\u201ctype\u201d:\u201c\u201d,\n\t\t}\n\t]\n}\n\nAfter Enrichment :\n{\n\n\t\u201cid\u201d: \u201caudit-log-1\u201d ,\n\t\u201cobjects\u201d: [\n\t\t{\n\t\t\t\u201cid\u201d:\u201c\u201d,\n\t\t\t\u201ctype\u201d:\u201c\u201d,\n\t\t}\n\t],\n\t\u201cenriched-objects\u201d :[\n\t\t{\n\t\t\t\u201cid\u201d:\u201c\u201d,\n\t\t\t\u201cname\u201d:\u201c\u201d,\n\t\t\t\u201ctype\u201d:\u201c\u201d\n\t\t}\n\t]\n}""}, {""user"": ""svummidi"", ""timestamp"": ""1673627782.441819"", ""content"": ""<@U0336QZAF98> - I think your approach works but I see two issues worth thinking about:\n1. Performance: For any indexing system update is expensive. Literally it need to read whole record, insert ( delete tombstone + add new), compact later. If there is a conflict between different services, it need to retry the whole. \n2. Eventual consistency\n3. Failures : It is an opinionated comment. If it is distributed design, we need to design for failures. \nEvery one from Skyhigh knows how enterprises uses audit logs. They ingest them to SIEM and Activity Monitoring systems. We need to provide an API for them as a feed once the product is operationalized. We don\u2019t want to make half baked entries visible.\n\nIf every create, update and delete API can return inserted/updated/deleted object, it gives enough context for app layer. In insert/update case there is no extra lookup cost because we already have reference to the updated object. We need to consider this pattern at least for future work, even if we use a different solution now. We fallback to external id resolution only if app can not figure out names without making additional calls.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673630939.731629"", ""content"": ""I guess I moved out of Skyhigh before customers started pulling out audit logs into their SEIMs or didn't realize they were doing so :slightly_smiling_face:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673632155.312099"", ""content"": ""1. Do we log failed states as well for POST/PUT/DELETE?\nI don't think we need to do this. The only failures of interest would be auth related.\n\n1. We need to think about way to log context in Async events.\nWe had a thread on this some time back (). Don't think it was resolved fully/correctly.\n\nIf an async request is fired and the the browser is closed immediately, apps would no longer be actively involved in the completion of that request and cannot create a record at the the end of the txn.\n\nIn these situations, typically completion of a transaction is tracked by a backend service. In such situations, we need the backend service to be responsible for firing the audit log request.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673632258.550799"", ""content"": ""If we define the source of audit log as the service that manages the transaction (as opposed to apps or backend) we'd have some sanity at least in the definition.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673634814.805119"", ""content"": ""Re: ES update approach -- considering the fact that we wouldn't have a huge number of writes and the fact that audit log for an event doesn't need to be immediately available like a live-log (we can define a reasonable SLA) we might be able to take it forward another way.\n\n1. Instead of writing partial records to the main index and updating, hold the partial records in some temporary storage or low-cost storage. We'd also need to remember the number of responses we need to get back to complete the record and the TTL (say 10 mins).\n2. As the partial records come in, we just append to the object and when the total responses outstanding becomes 0, we then merge this into a proper event and place it in the main queue to be sent to ES.\nThis solves two of the problems cited (a) updating the object in ES which is costly and may have an impact on ES cluster requirements & cost (b) returning partial objects to the customer, which can be a real issue\n\nThis obviously introduces more complexity into the system that is already complicated. Also merging could be a bit of messy problem, though it won't be unsolvable.\n\nStill, we don't solve the problem of missing context if say, an object is updated and immediately deleted. We might not be able to resolve the name of the object when the update event is picked up to be enriched, which takes us back to the original problem.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673635196.828549"", ""content"": ""One way to solve this problem could be to record the names of just the deleted entries in the DB and age the records out after some time. This avoids the risk of using just an in-memory cache with a TTL just in case the update queue is very slow.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673635273.768839"", ""content"": ""<@U033PPLNFRU> I don\u2019t think that we don\u2019t need to build a high level of attribution like SF because the objects that people manipulate are not as business critical in our system.\n\nWe can always capture 2 levels of action (high-level, and detailed) and display the high level.\n\nHaving said that I think we should not map login/logout to C, U, D. If we stick to 5 or 6 top level events and save off the second level event type we can move forward.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673635951.252719"", ""content"": ""It also looks like we need to store the IDs along with names until we get to some level of confidence in what we are producing. We'll only display names in the UI, but we should hold Type + IDs as well.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673849026.801269"", ""content"": ""Actually, if we have a system where we can record the names of the deleted entities, maybe we can stick to the existing implementation itself. One of the main reasons why we didn't want the the lambda or the service to resolve IDs to Names was the fear that we might not be able to resolve names of deleted entities.\n\nLet's say we build a store for deleted <{Tenant}:{Type}:{Id} -> Name> entries that is situated in each of the db schemas or centrally in DynamoDb which allows TTLs etc. (let's call it Deleted-Index)\n\nAs the Audit log records come in with Type+IDs, the lambda/service invokes the resolvers of the specific services to return the names (looks like the need for resolvers is a clear requirement). If the resolver comes back with a not-found response, we'll check with the Deleted-Index to get the name.\n\nIf we do this we can save ourselves from the need for asynchrony/patching etc.\n\nwdyt?""}, {""user"": ""aganivada"", ""timestamp"": ""1673850787.994699"", ""content"": ""makes sense <@U026PMDB1ND>, I think Dynamodb with TTL will be better so we don't have to worry about cleaning up the data.\n\nAnother option could be if we can store this data of deleted instances in elastic search itself and have a in elastic search to resolve this field during runtime, for example resolving ip to location in ES response this might be a bit complicated in ES but we will have one source where we have all the info.""}, {""user"": ""anair"", ""timestamp"": ""1674180895.377559"", ""content"": ""<@U02BV2DGUKC> <@U026PMDB1ND> if services is going to be easier than subjects- what are the services we currently have?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1674181024.310159"", ""content"": ""<@U033PPLNFRU> I don\u2019t think it will help the user if we call out services\u2026""}, {""user"": ""anair"", ""timestamp"": ""1674181617.296679"", ""content"": ""But if we put it under a meaningful name\n\u2022 Core Data = Team Administration\n\u2022 Pulse Manager = Pulses\n\u2022 User Actions = Actions\nBut the more I think about it, maybe this is overcomplicating a simple problem <@U026PMDB1ND>. Let us go very simple on Phase 1 then""}, {""user"": ""anair"", ""timestamp"": ""1674182140.441079"", ""content"": ""<@U026PMDB1ND> <@U02BV2DGUKC> is it ok to use CREATE, UPDATE, DELETE instead of POST, PATCH, DELETE?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1674182536.985639"", ""content"": ""Absolutely ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1674182571.216449"", ""content"": ""Using the latter set doesn\u2019t help the target users""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1674182721.618749"", ""content"": ""Instead of actual physical services we can call out logical service areas (pulse management, brokering, etc.) This might be useful for them when they search/group. We should also try and get the subject to the extent possible though we may not display the same""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1674182868.511709"", ""content"": ""<@U033PPLNFRU> so far apart from create, update, delete we have a few other events- login, logout, send pulse etc. I feel that we\u2019ll end up with ~10 in all""}, {""user"": ""anair"", ""timestamp"": ""1674182894.875479"", ""content"": ""<@U026PMDB1ND> I am actually currently simplifying the spec and the design. Send pulse is just a create?""}, {""user"": ""anair"", ""timestamp"": ""1674182923.706039"", ""content"": ""<@U026PMDB1ND> in your opinion should have a separate event and description column or just merge them?""}, {""user"": ""anair"", ""timestamp"": ""1674183056.019619"", ""content"": ""<@U026PMDB1ND> i am thinking of keeping it very simple with just CREATE, update, delete, login""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1674183509.634399"", ""content"": ""how do you get logout, change password etc.""}, {""user"": ""anair"", ""timestamp"": ""1674183534.727519"", ""content"": ""logout - part of login\nchange password - update""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1674183553.146659"", ""content"": ""I think force fitting things that don\u2019t make sense could make it wacky""}, {""user"": ""anair"", ""timestamp"": ""1674183611.592269"", ""content"": ""then we should stick to subjects (login, settings (for password), pulses, actions, etc)? wdyt <@U026PMDB1ND>""}, {""user"": ""anair"", ""timestamp"": ""1674183650.962429"", ""content"": ""<@U026PMDB1ND> customers much likely want to filter across subjects than create/update/delete""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1674183864.921529"", ""content"": ""<@U033PPLNFRU> it might be useful to check out audit log in Salesforce ( <@U028EDANJM9> - can you share creds of a dev instance), before finalizing on event types. Since we\u2019d lose the info at the time of generating the event, we may want to be very sure""}, {""user"": ""anair"", ""timestamp"": ""1674184094.447319"", ""content"": ""<@U026PMDB1ND> I looked at auth0, figma, jira, looking at SF rn. They are all very detailed and have so much attribution. Do we want to build anything near that? The idea was in phase 1 to build something to satisfy security reviews. In some long distant future we will have to build an actual robust audit log""}, {""user"": ""gshenoy"", ""timestamp"": ""1674186403.495619"", ""content"": ""SF dev instance <@U026PMDB1ND> <@U033PPLNFRU>\n\n\n / Axiamatic@123""}, {""user"": ""aganivada"", ""timestamp"": ""1674186891.051999"", ""content"": ""auth0 attribution might look developers oriented but they record entire request payload so they don't have to keep updating audit every time there is a new attribute. I think that might be helpful especially during security reviews. Also on Delete operation they don't give names they just give id's""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1674187198.206249"", ""content"": ""Yeah, I think SF is the best one for us to emulate both from the problem statement and target audience perspectives. \n\nAnother option is to carry two items - detailed action and action in the event - so that we can switch things at the presentation layer if we need extra details.\n\nAnything that requires us to modify the source of the event should be handled with care as we\u2019d have to make a lot of mods.\n\n\n\n(btw, sorry for the delay in responding). ""}, {""user"": ""gshenoy"", ""timestamp"": ""1674189003.871909"", ""content"": """"}, {""user"": ""rtaraniganty"", ""timestamp"": ""1674189169.594629"", ""content"": ""I think we can emulate the \u2018section\u2019 which is a bit like a logical service that we have been discussing ""}, {""user"": ""anair"", ""timestamp"": ""1674260153.844629"", ""content"": ""ack <@U026PMDB1ND> <@U02BV2DGUKC> shall put something together based on this discussion""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1674630547.385269"", ""content"": ""In the call tonight IST these are the items we need to finalize:\n\n1. Id to Name resolution, including deleted entity name resolution (where does it happen)\n2. Name resolver APIs in services\n3. Audit logs for Sync vs Async events\n4. Quick look at the proposed UX and any questions UI team might have\nWe can figure out timelines and integration over the next couple of days. Let's close out on these high level items today.\n\n <@U033PPLNFRU> ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1674630591.760769"", ""content"": "" is the page that details the requirements.""}]" "1680018842.813839 ","[{""user"": ""aganivada"", ""timestamp"": ""1680018842.813839"", ""content"": "" please review VPN overview and planning document\n\n\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680022307.400659"", ""content"": ""<@U02BV2DGUKC>\n\nThis is very well done. Appreciate the effort that went into building the page :clap:\n\nIn the existing arch;\n\n(1) Swagger label should probably on a different line?\n(2) RDS Access may need to be called SSH access. If possible we can color the line from Mac to bastion and bastion to RDS the same way to indicate how Mac accesses RDS.\n(3) May want to label less common icons\n(4) Can we indicate S2S token acquisition via Cognito?\n\n\nUpdated architecture\n\n(1) Indicate the auth token(s) involved on the lines left of API Gateway\n\nQuestions:\n\n(1): Eventbridge API calls to access service -> How can this become a security issue?""}, {""user"": ""aganivada"", ""timestamp"": ""1680063707.274449"", ""content"": ""Thank you <@U026PMDB1ND>, will update the document\n\n> Eventbridge API calls to access service -> How can this become a security issue?\nthis endpoint will be available to outside world (whitelisted without authorizer check) however it is protected with s2s token auth done by spring itself.""}]" "1689141351.400469 ","[{""user"": ""aganivada"", ""timestamp"": ""1689141351.400469"", ""content"": ""<@U03KLHDKL1H> I have a job scheduled to run once a day so it is running it at 5:30 am IST is there any way to trigger it now for testing without updating the schedule?""}, {""user"": ""askumar"", ""timestamp"": ""1689141575.035419"", ""content"": ""<@U02BV2DGUKC> not sure how to do it in int later.\n\nWhen deployment happens it triggers all jobs.""}, {""user"": ""aganivada"", ""timestamp"": ""1689143337.835729"", ""content"": ""ok, yeah I see other jobs running on startup but the backup for some reason ~failed~ didnt trigger""}, {""user"": ""aganivada"", ""timestamp"": ""1689143473.570389"", ""content"": ""ok got it there is an error in the job""}, {""user"": ""askumar"", ""timestamp"": ""1689143511.690429"", ""content"": ""Yeah it might be some error even before job could execute.""}, {""user"": ""aganivada"", ""timestamp"": ""1689147486.674459"", ""content"": ""<@U03KLHDKL1H> need to deploy job-executor in stage and prod, do we need latest cdk changes or can I deploy from main?""}, {""user"": ""askumar"", ""timestamp"": ""1689147550.328379"", ""content"": ""We will need the cdk changes , though there is just 1 line change if need to be cherrypicked""}, {""user"": ""askumar"", ""timestamp"": ""1689147616.955569"", ""content"": ""<@U02BV2DGUKC> I guess it can be deployed from develop as well, since there has been no other change""}, {""user"": ""aganivada"", ""timestamp"": ""1689147617.802999"", ""content"": ""hmmm I can also deploy from develop if the change is merged""}, {""user"": ""askumar"", ""timestamp"": ""1689147626.803509"", ""content"": ""Yeah it is merged""}, {""user"": ""aganivada"", ""timestamp"": ""1689147636.493269"", ""content"": ""let me try in stage""}, {""user"": ""aganivada"", ""timestamp"": ""1689147744.202549"", ""content"": ""azure account refresh changes are verified right? is there any key we need to add to deployment details or secrets manager?""}, {""user"": ""askumar"", ""timestamp"": ""1689147804.448939"", ""content"": ""No nothing else is required, it will be same as in INT which I already added yesterday""}, {""user"": ""askumar"", ""timestamp"": ""1689147822.556659"", ""content"": ""Changes are verified""}, {""user"": ""aganivada"", ""timestamp"": ""1689147851.491119"", ""content"": ""ok so no need to add any additional key""}, {""user"": ""askumar"", ""timestamp"": ""1689147862.144059"", ""content"": ""Yeah that's correct""}]" "1673523711.639469 ","[{""user"": ""askumar"", ""timestamp"": ""1673523711.639469"", ""content"": "" \nIs there a script being executed in PROD for the tenancy endpoint, API gateway is denying request due to malicious URL ?\n\n\""Request\"": \""GET /tenancy/api/v1/tenants/(25,%2014509%20%20%20%2014729%0AName:%2025,%20dtype:%20int64) HTTP/1.1\"",\n\n\""X-Axmgw-RequestId\"": \""eoCpnjVMPHcEJhA=\"",\n \""Referer\"": \""-\"",\n \""User-Agent\"": \""python-urllib3/1.26.6\"",\n \""X-Forwarded-Proto\"": \""http\"",\n \""X-Forwarded-Host\"": \""-\"",\n \""Forwarded\"": \""for=49.205.251.29;host=;proto=https\"",\n \""X-Forwarded-For\"": \""10.55.9.98\"",\n \""X-Axmgw-ClientIp\"": \""49.205.251.29\""\n }""}, {""user"": ""askumar"", ""timestamp"": ""1673524947.698699"", ""content"": ""Alert got triggered due to tenant clean up\nNo systemic issues.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673543139.277089"", ""content"": ""<@U02BV2DGUKC> - did you invoke anything like this?""}, {""user"": ""aganivada"", ""timestamp"": ""1673543149.896609"", ""content"": ""yes Rama""}, {""user"": ""aganivada"", ""timestamp"": ""1673543183.822929"", ""content"": ""I was testing with a script got the values wrong while parsing""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673543197.687129"", ""content"": ""I did a whois on 49.205.251.29 and got this :slightly_smiling_face:\n\n```organisation: ORG-BTPL4-AP\norg-name: Beam Telecom Pvt Ltd\ncountry: IN\naddress: ATRIA CONVERGENCE TECHNOLOGIES LTD.,\naddress: 8-2-618/1/2, Banjara Hills, Road No -11,\nphone: +91-9121212121\nfax-no: +91-4066666348\ne-mail: \nmnt-ref: APNIC-HM\nmnt-by: APNIC-HM\nlast-modified: 2020-07-13T12:56:20Z\nsource: APNIC```""}, {""user"": ""aganivada"", ""timestamp"": ""1673543202.433379"", ""content"": ""this is fixed now and we cleaned up some stale tenants""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673543228.843429"", ""content"": ""Narrowed it down to anybody in HYD :slightly_smiling_face:""}, {""user"": ""aganivada"", ""timestamp"": ""1673543233.412119"", ""content"": ""actcorp is my internet provider""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673543240.536209"", ""content"": ""Nice""}, {""user"": ""aganivada"", ""timestamp"": ""1673543294.642019"", ""content"": ""BTW earlier today evening Seshan also went full Sherlock on the ip :grinning:""}, {""user"": ""aganivada"", ""timestamp"": ""1673543625.547819"", ""content"": ""Looks like I really cant hide myself if I mess up something. Atleast for that reason I need VPN :sweat_smile:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673547585.549719"", ""content"": ""We should always investigate anything that looks suspicious. At least the not-so-sophisticated hacks can be caught early that way.""}]" "1693410195.545539 ","[{""user"": ""svummidi"", ""timestamp"": ""1693410195.545539"", ""content"": ""<@U02BV2DGUKC> - I don\u2019t see any pulse for this week retro. But I can see open pulse and my name is listed. I tried to nudge but I noticed the below error in logs\n`No message history found for user U05D3HNDW3D and instance id 130578 - not nudging`\nI noticed similar error few other users also, please confirm if you received the pulse or not.""}, {""user"": ""aganivada"", ""timestamp"": ""1693410283.772809"", ""content"": ""<@U040RCBPBEC> I received the nudge""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693410288.731109"", ""content"": ""Didn\u2019t receive a pulse. ""}, {""user"": ""aganivada"", ""timestamp"": ""1693410400.976529"", ""content"": ""this is weird, <@U03RSS0S76Y> could this be TZ issue?""}, {""user"": ""aganivada"", ""timestamp"": ""1693410406.813029"", ""content"": ""local TZ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693410412.071689"", ""content"": ""Last pulse received was on Aug 23. Wonder if there\u2019s an issue with a user in the list we bail and don\u2019t send to the remaining users in the list.""}, {""user"": ""aganivada"", ""timestamp"": ""1693410430.655509"", ""content"": ""yeah possible""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693410444.501069"", ""content"": ""<@U02BV2DGUKC> can you nudge individual users?""}, {""user"": ""aganivada"", ""timestamp"": ""1693410469.383799"", ""content"": ""sure <@U026PMDB1ND>""}, {""user"": ""aganivada"", ""timestamp"": ""1693410540.279959"", ""content"": ""it says nudged successfully for all users, didnt get any error message :disappointed:""}, {""user"": ""aganivada"", ""timestamp"": ""1693410548.907229"", ""content"": ""Also got the nudge for my account""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693410563.283949"", ""content"": ""Because we asked all the teams to run retros here someone sabotaged ours? :grinning: ""}, {""user"": ""aganivada"", ""timestamp"": ""1693410676.782589"", ""content"": ""also I don't see <@U05PCBD9SHE>\u2019s name in respondents. <@U03NZ7Z52S2> should we run any API? I added Vishal as primary user last week was assuming we'd automatically send pulse to Vishal from this week""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693410719.033119"", ""content"": ""Still didn\u2019t get any nudge.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693410784.401499"", ""content"": ""We can probably postpone this by 12 hrs (or so) if we can\u2019t send out the pulses. Also, this could be a concerning development re: hf""}, {""user"": ""aganivada"", ""timestamp"": ""1693410837.600019"", ""content"": ""<@U026PMDB1ND> most likely this is TZ issue.""}, {""user"": ""aganivada"", ""timestamp"": ""1693410874.066479"", ""content"": "" can we update tz of users and nudge again?""}, {""user"": ""aganivada"", ""timestamp"": ""1693410893.341459"", ""content"": ""looks like users in PST TZ are not receiving pulses & nudges""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1693410926.116229"", ""content"": ""<@U02BV2DGUKC> let me investigate once and revert back , <@U03RSS0S76Y> can you make a pass to see if there is anything timezone specific?""}, {""user"": ""snangia"", ""timestamp"": ""1693410937.578599"", ""content"": ""<@U02BV2DGUKC> <@U026PMDB1ND> let me check""}, {""user"": ""snangia"", ""timestamp"": ""1693410978.985539"", ""content"": ""<@U02D4DUKDQC> I think it must be tz specific, checking""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1693410996.958789"", ""content"": ""<@U02BV2DGUKC> what is the tenantID? and is this running on stage or prod?""}, {""user"": ""snangia"", ""timestamp"": ""1693411018.716749"", ""content"": ""15328\nstage""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693411024.251249"", ""content"": ""PST TZ + newly added users, @anil?""}, {""user"": ""aganivada"", ""timestamp"": ""1693411029.491509"", ""content"": ""> We can probably postpone this by 12 hrs (or so) if we can\u2019t send out the pulses.\n<@U026PMDB1ND> may be we can join the call and decide? tmrw morning back-2-back meetings until 11 am IST""}, {""user"": ""snangia"", ""timestamp"": ""1693411042.386649"", ""content"": ""newly added user is a known issue <@U026PMDB1ND> for recurring + tz pulses""}, {""user"": ""aganivada"", ""timestamp"": ""1693411076.471079"", ""content"": ""<@U03RSS0S76Y> can you point me to any API we can run to fix the newuser issue?""}, {""user"": ""snangia"", ""timestamp"": ""1693411180.870619"", ""content"": ""<@U02BV2DGUKC> should fix for new user, but again delivery is dependent on new user tz""}, {""user"": ""snangia"", ""timestamp"": ""1693411238.688149"", ""content"": ""<@U02BV2DGUKC> what was the org name for this pulse?""}, {""user"": ""aganivada"", ""timestamp"": ""1693411255.628829"", ""content"": ""axm-survey-stage""}, {""user"": ""svummidi"", ""timestamp"": ""1693411445.563569"", ""content"": ""If you want to fill the survey weblinks""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1693411930.149109"", ""content"": ""<@U03RSS0S76Y> is this the correct pulse survey instance ID for investigation 130578 ?""}, {""user"": ""snangia"", ""timestamp"": ""1693412019.290499"", ""content"": ""<@U02D4DUKDQC> it's not yet 10am in PST and pulse is scheduled for 10am""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1693412113.082869"", ""content"": ""yes that seems correct <@U03RSS0S76Y> so it is working as expected\n\n```select email, tz, starts_at, started_at from svc_pulse_manager_default_axm.pulse_survey_user_instance psui where psui.pulse_survey_instance_id = 130578;\n\nemail,tz,starts_at,started_at\n,Asia/Kolkata,2023-08-29 04:30:00.000000 +00:00,2023-08-30 05:04:16.936153 +00:00\n,Asia/Kolkata,2023-08-29 04:30:00.000000 +00:00,2023-08-30 05:04:16.936153 +00:00\n,Asia/Kolkata,2023-08-29 04:30:00.000000 +00:00,2023-08-30 05:04:16.936153 +00:00\n,America/Los_Angeles,2023-08-29 17:00:00.000000 +00:00,\n,America/New_York,2023-08-29 14:00:00.000000 +00:00,\n,America/Los_Angeles,2023-08-29 17:00:00.000000 +00:00,\n,Asia/Kolkata,2023-08-29 04:30:00.000000 +00:00,2023-08-30 05:04:16.936153 +00:00\n,Asia/Kolkata,2023-08-29 04:30:00.000000 +00:00,2023-08-30 05:04:16.936153 +00:00\n,Asia/Kolkata,2023-08-29 04:30:00.000000 +00:00,2023-08-30 05:04:16.936153 +00:00\n,Asia/Kolkata,2023-08-29 04:30:00.000000 +00:00,2023-08-30 05:04:16.936153 +00:00\n,Asia/Kolkata,2023-08-29 04:30:00.000000 +00:00,2023-08-30 05:04:16.936153 +00:00\n,America/Los_Angeles,2023-08-29 17:00:00.000000 +00:00,```""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1693412149.933809"", ""content"": ""Q""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1693412303.926479"", ""content"": ""The issue we need to check is why Vishal did not get it, <@U02BV2DGUKC> did you run the API that sagarika pointed?""}, {""user"": ""snangia"", ""timestamp"": ""1693412335.271429"", ""content"": ""<@U02D4DUKDQC> that's a known issue, there's an existing tracker for it, will add it to this thread for reference""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1693412373.487099"", ""content"": ""sure thanks , we can pick that up in this sprint if needed""}, {""user"": ""snangia"", ""timestamp"": ""1693414055.051909"", ""content"": """"}, {""user"": ""hchintamreddy"", ""timestamp"": ""1693416771.577499"", ""content"": ""<@U026PMDB1ND> <@U040RCBPBEC> You should\u2019ve received the pulse now on slack can you confirm""}, {""user"": ""aganivada"", ""timestamp"": ""1693416844.003299"", ""content"": ""<@U02D4DUKDQC> for now we used the web link to submit pulses, we will close this recurring pulse and open a new one""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693416868.970529"", ""content"": ""I received now, but I had also submitted the pulse via web-link that Satya had generated. Shouldn\u2019t that have stopped the nudge?""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1693416911.504009"", ""content"": ""<@U03RSS0S76Y> Artem did not get the pulse though he is 3 hours ahead of PST we probably need to debug this\n\nAlso in IST the pulses were delivered 4 minutes but in PST it was 14 minutes after scheduled time""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1693416978.058109"", ""content"": ""<@U026PMDB1ND> I think the weblink is a hack and we do not have a defined behaviour for that as of now but we can fix that if needed""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693417220.740079"", ""content"": ""Maybe we should fix in 9.16. Just to not leave a hole in the process.""}, {""user"": ""snangia"", ""timestamp"": ""1693417955.174369"", ""content"": ""<@U02D4DUKDQC> I see scheduler job not being triggered by event bridge.\nat 22:43 IST I executed using the swagger, that must have delivered PST pulses""}, {""user"": ""snangia"", ""timestamp"": ""1693417971.746129"", ""content"": ""<@U02BV2DGUKC> is the event bridge rule active on stage?""}, {""user"": ""aganivada"", ""timestamp"": ""1693419604.762609"", ""content"": ""We fixed all the event bridges before deploying 0.9.14 <@U03RSS0S76Y> . I can cross check tmrw, <@U04JT69T00K> do we have the event bridge alert deployed on int?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693421916.442159"", ""content"": ""<@U02BV2DGUKC> <@U04JT69T00K> - did we fix things only in stage or in prod as well? Do we need to verify if things are working okay in prod?\n\n<@U02D4DUKDQC> - can you please check if we are delivering okay in prod?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693422404.378969"", ""content"": "" - logged to address the web submission's impact on subsequent nudges etc.""}, {""user"": ""aganivada"", ""timestamp"": ""1693459228.314359"", ""content"": ""> did we fix things only in stage or in prod as well? Do we need to verify if things are working okay in prod?\nwe didn't make any specifc changes in prod <@U026PMDB1ND> once we fixed stage we observed event bridge failures for a week or so before we publish to prod.\n\n> is the event bridge rule active on stage?\n<@U03RSS0S76Y> I see 2 event bridge rules failing in stage batch one is expected checking on pulse-delivery""}, {""user"": ""snangia"", ""timestamp"": ""1693459228.935959"", ""content"": ""<@U02BV2DGUKC> it stopped working yesterday again""}, {""user"": ""aganivada"", ""timestamp"": ""1693459308.834239"", ""content"": ""it says inactive and de-authorized""}, {""user"": ""aganivada"", ""timestamp"": ""1693459319.945759"", ""content"": ""<@U03RSS0S76Y> was it working before and stopped now?""}, {""user"": ""aganivada"", ""timestamp"": ""1693459333.519649"", ""content"": ""wondering how we got the pulses yesterday morning IST""}, {""user"": ""aganivada"", ""timestamp"": ""1693459384.382909"", ""content"": ""Also I see endpoint whitelisted""}, {""user"": ""snangia"", ""timestamp"": ""1693459385.177359"", ""content"": ""yes I see last one at 13:07 yesterday""}, {""user"": ""snangia"", ""timestamp"": ""1693459491.093129"", ""content"": """"}, {""user"": ""snangia"", ""timestamp"": ""1693459516.905929"", ""content"": ""looks like after 10:49 it stopped for some while, and then triggered for last time at 13:07""}, {""user"": ""aganivada"", ""timestamp"": ""1693459554.451149"", ""content"": ""hmmm based on event bridge it looks like app returned 401 so they stopped querying""}, {""user"": ""snangia"", ""timestamp"": ""1693459569.261819"", ""content"": ""even the cadence is not proper around ~10:49~ since a while.""}, {""user"": ""snangia"", ""timestamp"": ""1693459574.042809"", ""content"": ""i see 200 in logs""}, {""user"": ""aganivada"", ""timestamp"": ""1693459841.947539"", ""content"": ""I don't see any issue with VPN related settings for some reason event bridge got unauthorized response from backend and then they stopped the rule""}, {""user"": ""snangia"", ""timestamp"": ""1693459852.287859"", ""content"": ""looks like problem started around 26 Aug""}, {""user"": ""pjha"", ""timestamp"": ""1693460185.134549"", ""content"": ""Production, 5 events are failing c.c <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1693460359.177889"", ""content"": ""<@U04JT69T00K> can you check on the connections failing in prod?""}, {""user"": ""aganivada"", ""timestamp"": ""1693460466.083229"", ""content"": ""<@U03RSS0S76Y> <@U02D4DUKDQC> can we have a quick call? want to check on the root cause of this and fix failing event bridges in prod""}, {""user"": ""pjha"", ""timestamp"": ""1693460508.184349"", ""content"": ""<@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1693461973.054979"", ""content"": ""<@U02D4DUKDQC> /<@U026PMDB1ND> Looks like there was an error from AWS while fetching token from cognito in both stage and prod and these are random failures because a lot of other connections seem to work fine. During my last ticket with AWS they said they have to debug a failing connection and we destroyed all failing connections last time to fix the issue. For prod is there any connection that is ok to leave for AWS to debug so we can redeploy the other trigger connections?""}, {""user"": ""aganivada"", ""timestamp"": ""1693463370.509769"", ""content"": ""<@U04JT69T00K> can we turn on the alert in prod for failing event bridge rules once we fix the issues?""}, {""user"": ""aganivada"", ""timestamp"": ""1693464586.081599"", ""content"": ""<@U04JT69T00K> can we check failures in prod now? user action nudge we will keep it for now so AWS can debug. discovery batch is expected""}, {""user"": ""pjha"", ""timestamp"": ""1693464832.894169"", ""content"": """"}, {""user"": ""pjha"", ""timestamp"": ""1693464880.126489"", ""content"": ""production-trigger-pulse-delivery-event-api-rule and\nproduction-plat-epoch-sfn-change-event-rule-rule are failing""}, {""user"": ""aganivada"", ""timestamp"": ""1693465112.423849"", ""content"": ""pulse delivery should be fixed now""}, {""user"": ""pjha"", ""timestamp"": ""1693465300.101409"", ""content"": ""last failure for pulse-delivery was around 12""}, {""user"": ""aganivada"", ""timestamp"": ""1693465420.239319"", ""content"": ""yeah that makes sense so only plat-epoch-sfn-change-event-rule-rule is failing now?""}, {""user"": ""pjha"", ""timestamp"": ""1693465524.746179"", ""content"": ""There seems to be no failure after 12:25""}, {""user"": ""pjha"", ""timestamp"": ""1693465797.080419"", ""content"": ""Did we remove 'production-plat-epoch' ?""}, {""user"": ""aganivada"", ""timestamp"": ""1693465876.595559"", ""content"": ""hmmm no I just redeployed *#trigger-pulse-delivery-destination* and *#trigger-pulse-auto-extension*""}, {""user"": ""aganivada"", ""timestamp"": ""1693466011.399089"", ""content"": ""this rule may not have API destination""}, {""user"": ""aganivada"", ""timestamp"": ""1693466126.617919"", ""content"": ""destination is a queue <@U04JT69T00K>, I think we can debug this later""}, {""user"": ""pjha"", ""timestamp"": ""1693466166.118639"", ""content"": ""<@U02BV2DGUKC> yes, I can see in the rules, Last failure is at 10:55""}, {""user"": ""aganivada"", ""timestamp"": ""1693486731.884619"", ""content"": ""<@U04JT69T00K> did we deploy the alert in prod?""}, {""user"": ""pjha"", ""timestamp"": ""1693486825.917479"", ""content"": ""<@U02BV2DGUKC>, yes I have deployed alert in prod .""}, {""user"": ""pjha"", ""timestamp"": ""1693486961.069649"", ""content"": ""Created alert for all the event as well as separate for pulse-delivery and auto-extention""}, {""user"": ""aganivada"", ""timestamp"": ""1693487017.034139"", ""content"": ""cool thank you I will add runbook for the alert""}, {""user"": ""aganivada"", ""timestamp"": ""1693487109.978109"", ""content"": ""<@U04JT69T00K> can you share rule name for the other 2 alerts?""}, {""user"": ""pjha"", ""timestamp"": ""1693487212.286529"", ""content"": ""\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693494459.288729"", ""content"": ""<@U02BV2DGUKC> has this been working since redeployment?""}, {""user"": ""aganivada"", ""timestamp"": ""1693495747.913999"", ""content"": ""yes <@U026PMDB1ND> after redeployment event bridges are working fine""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693496156.189269"", ""content"": ""<@U02BV2DGUKC> - I suppose you also opened a ticket with AWS as well""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693496193.326539"", ""content"": ""Also, is the alert situation straightened out?""}, {""user"": ""aganivada"", ""timestamp"": ""1693496204.031129"", ""content"": ""yes <@U026PMDB1ND>, we raised it from production account no update as of now""}, {""user"": ""aganivada"", ""timestamp"": ""1693800037.674779"", ""content"": ""<@U03RSS0S76Y> <@U04JT69T00K> <@U02D4DUKDQC> <@U026PMDB1ND>\n\nQuick update on this thread, we got an update from AWS Friday evening that they are still investigating on why the connection went to de-authorized state. Unfortunately when a connection goes to de-authorized state they remove creds from the connection so re-attempt will never succeed. Meanwhile, they have recommended us to update the secret back to recover the connection instead of destroying and deploying I tried the suggested approach on int and stage and we could recover the connection back.\n\nI will update this thread again once we hear from AWS.""}, {""user"": ""aganivada"", ""timestamp"": ""1694490944.574959"", ""content"": ""Update on this issue form AWS,\n\n```We noticed that the ApiDestination with connection arn:aws:events:us-west-2:643306803378:connection/production-S2S-Token-trigger-user-action-nudge-Connection/5ebb9fcd-f476-4258-9f5f-c07b2548209c started to fail consistently starting 2023-08-28T12:20:59.224Z. The exception returned was a non retriable exception that is returned when the ApiDestination resource is inactive. The state change of the ApiDestinations resource happens when the connection associated with it is deauthorized.\n\nFurther, upon checking few internal logs, our team noticed that each of the requests to the OAuth endpoint to get credentials is taking longer than the 5 second timeout which resulted the connection to be changed to a DEAUTHORIZED state. \n\nNext plan of action:\n---------------------\n\u25ba Update the connection with the same configuration as what you currently have. This will trigger another attempt at authorization for the connection. Or you can create a new connection and associate it with your API Destination.\n\u25ba As for why the connection is moving to de-authorized, kindly look into your OAuth workflow to see why the response time for requests is consistently going above the 5 second timeout.\n---------------------```\nI still think they haven't found the root cause since this is a direct integration between event bridge connections and cognito. Also I didn't find any clear instructions on debugging why cognito would take more than 5 seconds to generate token. if that is the case all of our backend integration should also be impacted. I replied with following comment, please let me know if there is anything we can add\n\n```We are currently using AWS cognito for oauth and almost all services reach cognito to generate tokens, I dont see any option in AWS console to debug on why cognito oauth endpoint is taking more than 5 seconds. Can you kindly help sharing metrics or logs from our cognito userpool linked to the failing connection . Since we use cognito for all major integrations please let us know if there are any suggested settings to help improve performance.\n\nYou also mentioned that there were some non-retriable exceptions causing the resource state to be inactive, assuming oauth connection is working how can we recover resource back to active? with the help of instructions you shared earlier we have a way to authroize a connection flagged as de-authorized but I am still not clear on how we can update status of a resource (API destination) back to active.```\n""}]" "1682424984.011519 ","[{""user"": ""pjha"", ""timestamp"": ""1682424984.011519"", ""content"": ""<@U02BV2DGUKC> <@U028EDANJM9> <@U026PMDB1ND> please review Querying CloudFront Logs""}, {""user"": ""aganivada"", ""timestamp"": ""1682490380.122779"", ""content"": ""thank you <@U04JT69T00K>, can we create Athena tables in all 3 env's and link to the bucket? We can then move this to how-to article on querying cloudfront access logs""}, {""user"": ""pjha"", ""timestamp"": ""1682494777.476719"", ""content"": ""Int and Stage is done, will create in production once changes are deployed.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682532456.567849"", ""content"": ""<@U04JT69T00K> - Also is the entire provisioning of this done via CDK or were there manual steps involved?\n\nlooks like we have minor spelling issue in the DB name. Let's fix that and this page with the correct name.""}]" "1684232022.734789 ","[{""user"": ""aganivada"", ""timestamp"": ""1684232022.734789"", ""content"": ""<@U02SCRTM2M7> deploying vault-access service with CDK changes of the manual changewe did to fix the auth issue other day. Will verify report generation after deployment and let you know. cc: <@U02HCMTQU3W>""}, {""user"": ""aganivada"", ""timestamp"": ""1684245411.520779"", ""content"": ""<@U02SCRTM2M7> reports seems to be working after deployment. Please let me know if you notice any issues.\n\n<@U02HCMTQU3W> are we using vault in any other flow where we need to whitelist oauth related endpoints? may be web-app flow?""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1684245607.564849"", ""content"": ""<@U02BV2DGUKC> we do use it for web surveys, will check and update.\nI guess we also use this in jira web hooks <@U028EDANJM9> can you please check once ?""}, {""user"": ""aganivada"", ""timestamp"": ""1684251478.989379"", ""content"": ""<@U02HCMTQU3W> can we get this list from vault service?""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1684252163.007019"", ""content"": ""yes we can- ""}, {""user"": ""aganivada"", ""timestamp"": ""1684252432.957479"", ""content"": ""cool thank you <@U02HCMTQU3W> will whitelist `survey` and `jira-webhook`""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1684256449.076969"", ""content"": ""Thanks <@U02BV2DGUKC>, will test it once you confirm""}, {""user"": ""aganivada"", ""timestamp"": ""1684257483.179419"", ""content"": ""changes are deployed <@U02HCMTQU3W> please verify whenever you get a chance and let me know if we run into any issues.""}, {""user"": ""aganivada"", ""timestamp"": ""1684257642.330289"", ""content"": "" can we include regression tests for reporting, web-app survey & jira-webhook during 0.9.11? basic sanity should be sufficient we are making some changes to vault-acess service for VPN.""}, {""user"": ""araman"", ""timestamp"": ""1684258357.849129"", ""content"": ""Sure <@U02BV2DGUKC> Will take up in next sprint.""}, {""user"": ""aganivada"", ""timestamp"": ""1684258550.234569"", ""content"": ""thank you <@U03DHUAJVMK>""}, {""user"": ""bganganna"", ""timestamp"": ""1684288478.299999"", ""content"": ""<@U02BV2DGUKC> ST are already there for these. jira ST is failing and <@U034RLJA97X> is looking into it.\nTests/DashboardApp/pdf_report.robot\nTests/PulsesurveyApp/webapp_survey.robot\nTests/Jira/jira_bidirectional_sync.robot""}, {""user"": ""bganganna"", ""timestamp"": ""1684288636.508799"", ""content"": ""Not sure as part of webonly survey ST, lambda invocation is tested . Will check on this""}, {""user"": ""aganivada"", ""timestamp"": ""1684294035.768469"", ""content"": ""sure thank you <@U03NZ7Z52S2>""}, {""user"": ""sranjan"", ""timestamp"": ""1684318210.651089"", ""content"": ""cc <@U028EDANJM9>""}, {""user"": ""sranjan"", ""timestamp"": ""1684320137.661019"", ""content"": ""<@U02BV2DGUKC> <@U03NZ7Z52S2> <@U028EDANJM9> Tests/Jira/jira_bidirectional_sync.robot is passing on my local system .""}, {""user"": ""aganivada"", ""timestamp"": ""1684320959.157099"", ""content"": ""sorry <@U034RLJA97X> what is the context, is this state expected? we whitelisted /jira-webhook from vault-access WRT VPN changes so you may notice access to vault for all other endpoints will be restricted to VPN-only while this endpoint is open for consumption since we need it for oauth validation""}, {""user"": ""bganganna"", ""timestamp"": ""1684321047.971719"", ""content"": ""<@U034RLJA97X> its failing with No keyword with name 'Oauth2 Get jira Authorization Url' found.""}]" "1689623321.990129 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1689623321.990129"", ""content"": ""<#C028U328HMG|engg> makes sense. I was thinking on-call, but this is okay. Is there way to subscribe to multiple channels?""}, {""user"": ""aganivada"", ""timestamp"": ""1689656405.281339"", ""content"": ""added integration to both <#C028U328HMG|engg> & <#C03KDBFCMM2|on-call>. subscription will be managed by cc: <@U04JT69T00K>""}]" "1680242029.986749 ","[{""user"": ""ppant"", ""timestamp"": ""1680242029.986749"", ""content"": ""<@U026PMDB1ND> When we use specific versions in Poetry, there are some vulnerability checks which fail for `py==1.11.0` in Python 3.8 which we use in our Gitlab pipelines (). This package is an indirect dependency and also its the latest version of `py`. But updating Python to 3.10.7 in pipelines fix the security check. Shall I update the Python version for all lambda repos?""}, {""user"": ""ppant"", ""timestamp"": ""1680242251.450889"", ""content"": ""This comes from pytest""}, {""user"": ""aganivada"", ""timestamp"": ""1680243171.549829"", ""content"": ""<@U0431DZTPJM> is this a warning or error?""}, {""user"": ""aganivada"", ""timestamp"": ""1680243213.736519"", ""content"": ""I think we should be ok updating the version since we are anyways going to have a regression test on lambdas""}, {""user"": ""ppant"", ""timestamp"": ""1680243348.636719"", ""content"": ""<@U02BV2DGUKC> Its an error. The pipeline fails if it finds a vulnerability with any package\u2019s version""}, {""user"": ""ppant"", ""timestamp"": ""1680244062.535499"", ""content"": ""The other option is we can remove safety check from pipeline :slightly_smiling_face:""}, {""user"": ""aganivada"", ""timestamp"": ""1680244196.643849"", ""content"": ""Ok :) can we update python version and test with one lambda for any potential issues and then update all others?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680244769.046149"", ""content"": ""Does AWS support 3.10?""}, {""user"": ""ppant"", ""timestamp"": ""1680244871.095809"", ""content"": ""Apparently no. They support 3.7 to 3.9 ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680244925.560369"", ""content"": ""Yeah. I vaguely remember trying the 3.10 route once only to realize that aws lambda has no support :man-facepalming: ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680244967.751929"", ""content"": ""What is minimally needed to be both secure and pass the build?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680245046.842419"", ""content"": ""Is there a pytest that works with 3.8 or 3.9 ?""}, {""user"": ""ppant"", ""timestamp"": ""1680245317.451709"", ""content"": ""Yes it was working before. I will need to try some combinations to check which version it was and which is compatabile with safety""}, {""user"": ""ppant"", ""timestamp"": ""1680252750.276019"", ""content"": ""The pipeline is fixed now with security checks. Turns out the py package was cached in gitlab pipeline as we use pip\u2019s cache in CI `PIP_CACHE_DIR: \""$CI_PROJECT_DIR/.cache/pip\""`\nThese are the versions that work and will be using them in all lambdas\n```bandit = \""1.7.5\""\nboto3 = \""1.26.93\""\nblack = { version = \""23.1.0\"", allow-prereleases = true }\ncoverage = \""7.2.2\""\nflake8 = \""5.0.4\""\nipython = \""8.11.0\""\nisort = { version = \""5.12.0\"", extras = [\""pyproject\""] }\njinja2-cli = \""0.8.2\""\nmkdocs = \""1.3.0\""\nmkdocstrings = \""0.20.0\""\nmkdocs-material = \""8.5.4\""\nsafety = \""2.3.4\""\npylint = \""2.17.0\""\npytest = \""7.2.2\""\npytest-cov = \""4.0.0\""\npytest-sugar = \""0.9.6\""\npytest-xdist = \""3.2.1\""```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680282573.053529"", ""content"": ""Thanks!""}]" "1682556071.572599 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1682556071.572599"", ""content"": ""<@U02BV2DGUKC> <@U04JT69T00K> - GitLab charges us $10 for 1000 CICD minutes.\n\nFor us to justify $60 of our own runner we need to move 6000 minutes of workload to that runner. If we can move all of automated-tests-v1 and a couple of other costly projects off shared runners we'll get enough benefit.\n\n""}, {""user"": ""aganivada"", ""timestamp"": ""1682567228.249839"", ""content"": ""cool thank you for the details <@U026PMDB1ND>, we can disable shared runner once we are done with 0.9.10 validation this week. Prashant also found a way to turn on GL-runner on-demand we will attempt a POC this sprint""}]" "1675178713.484429 ","[{""user"": ""rvaidya"", ""timestamp"": ""1675178713.484429"", ""content"": ""<@U02BV2DGUKC> <@U0431DZTPJM>\n\n\nFew questions on the above logs :\n1. If the ProductAdmin logs in without any permission in the permission set , core is throwing AccessDenied with 500. Shouldn;t it return 403 or something else? We cant throw 500 to the frontend.\n2. Discovery is still returning the Products cos we are querying /pvt apis. And so some of the summary stats are getting returned from the API on the apps. The PID/Product Admin preauthorize check is not present in PVT table? I thought we had this.\nThis is with ref to \n\ncc ""}, {""user"": ""aganivada"", ""timestamp"": ""1675219500.576019"", ""content"": ""<@U03KLHDKL1H> can you take a look at (1)\n\n<@U02GC8SE18V> for (2) we have a filter on getAllPVTs call in core do we need to add this check elsewhere ?""}, {""user"": ""aganivada"", ""timestamp"": ""1675225983.136469"", ""content"": ""<@U03KLHDKL1H>, <@U0336QZAF98> has an idea how we can fix (1)\n\n> we just have to define globakexceptionhandler which when accessdenied exception happens it return 403 instead of 500\nif there is a platform ticket already can we address it otherwise we can create a new ticket and work on this. changes should be in axm-commons I guess. cc: <@U02GC8SE18V>""}, {""user"": ""askumar"", ""timestamp"": ""1675262244.642199"", ""content"": ""<@U02GC8SE18V> for 2nd I have tried looking into the paths for the PVT and product instance API, the validation is same for both the calls, so couldn't find anything there.\nWould it be possible to get the exact request with token that is coming to core please?\n<@U02BV2DGUKC> Added for global exception handler""}, {""user"": ""rvaidya"", ""timestamp"": ""1675263223.314899"", ""content"": ""<@U03KLHDKL1H> the user details are present in the bug. Will that help ?""}, {""user"": ""askumar"", ""timestamp"": ""1675263575.718199"", ""content"": ""I tried <@U02GC8SE18V> but couldn't really find anything so far, so looking for something that can help identify the issue..\nWill keep debugging""}, {""user"": ""rvaidya"", ""timestamp"": ""1675263589.404059"", ""content"": ""Also <@U02BV2DGUKC> <@U03KLHDKL1H> can we enable debug logs to check the headers/token is coming fine?""}, {""user"": ""askumar"", ""timestamp"": ""1675264823.431799"", ""content"": ""Enabling debug logs in stage for core-data.""}, {""user"": ""askumar"", ""timestamp"": ""1675267084.355399"", ""content"": ""Token claims are coming in debug logs correctly :\n""}, {""user"": ""rvaidya"", ""timestamp"": ""1675267882.394759"", ""content"": ""what about this log : X-Custom-Token-Header empty or not set, skipping build user context""}, {""user"": ""askumar"", ""timestamp"": ""1675268128.648369"", ""content"": ""right <@U02GC8SE18V> this is the token that we were trying to check in morning locally, I could see it working locally...need to investigate the difference""}, {""user"": ""askumar"", ""timestamp"": ""1675323808.576439"", ""content"": ""Seems like we have an issue with the permission mapping and check that is causing the issue.\nWe have a check in core addAuthorizationV2FilterForProductInstance\nhere we check that if permission are null then we say that it is a backend call so no authorization applied.\n\nHowever when we look at the place in commons libs\n\nHere when UserContextInterceptor is reading the permission it checks that permissions should be neither Empty nor NULL.\nIn our case the permission in token is empty list, so the interceptor does not set the permissions and it defaults to NULL.\nNow when this permission in read by addAuthorizationV2FilterForProductInstance , the NULL permission means that it is a backend call and no authorization is applied on it. Hence the issue.\ncc <@U02GC8SE18V> <@U02BV2DGUKC> <@U0336QZAF98>\n<@U03DHUAJVMK> could you please let me know if ProductAdmin having no product permission a regular use case?""}, {""user"": ""aganivada"", ""timestamp"": ""1675326080.616919"", ""content"": ""thank you <@U03KLHDKL1H> <@U0336QZAF98> this is a very good find when we made this change initially we didn't anticipate non orgadmin roles to have empty permissions. though we are working on a UI fix I think we should also fix this from backend.\n\n<@U02GC8SE18V> do we want to move to plat or shall we create a new HF ticket for this?""}, {""user"": ""rvaidya"", ""timestamp"": ""1675330949.666339"", ""content"": ""Lets move it to PLAT, let me do that. <@U02BV2DGUKC> <@U03KLHDKL1H>""}, {""user"": ""rvaidya"", ""timestamp"": ""1675331262.507009"", ""content"": ""<@U03KLHDKL1H> ""}, {""user"": ""askumar"", ""timestamp"": ""1675341672.009599"", ""content"": ""<@U02GC8SE18V> <@U03DHUAJVMK> we have deployed the fix\n to STAGE\nI tested the change with the given users in ticket\nOnly the orgAdmin with product permission can see it and productAmdin can't see the products for the given test account.\n\n<@U03DHUAJVMK> <@U03NZ7Z52S2> May I please ask you to validate this on your side as well?\ncc <@U02BV2DGUKC> <@U0336QZAF98>""}, {""user"": ""aganivada"", ""timestamp"": ""1675350898.875749"", ""content"": ""<@U03KLHDKL1H> I hope we updated stage.manifest.txt of release branch for core-data""}, {""user"": ""askumar"", ""timestamp"": ""1675350985.862129"", ""content"": ""Yeah <@U02BV2DGUKC> Guru updated it""}, {""user"": ""aganivada"", ""timestamp"": ""1675351366.754499"", ""content"": ""awesome thank you!!""}]" "1687551279.176689 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1687551279.176689"", ""content"": "" - did we change some nw/sg config that would disallow connections to RDS via sshuttle?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687552261.945949"", ""content"": ""Did we change something in the Perimeter 81 config?""}, {""user"": ""ppant"", ""timestamp"": ""1687558619.241799"", ""content"": ""Not sure about RDS but yesterday we added P81\u2019s security group to Aurora which would allow connections to Aurora with just VPN (no sshuttle)""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687558634.999909"", ""content"": ""Hmm.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687558657.593839"", ""content"": ""It doesn't seem to be working as expected then""}]" "1684243688.403699 ","[{""user"": ""aganivada"", ""timestamp"": ""1684243688.403699"", ""content"": ""<@U0431DZTPJM> there is a build failure in develop after I merged HF changes in tenancy I am looking into it, will update you shortly""}, {""user"": ""ppant"", ""timestamp"": ""1684243772.624189"", ""content"": ""Faced such a failure today also in develop. Some random tests fail but locally they all pass. If you retrigger the pipeline, it will succeed""}, {""user"": ""ppant"", ""timestamp"": ""1684243784.708179"", ""content"": ""I guess it has to do with parallel execution of tests?""}, {""user"": ""aganivada"", ""timestamp"": ""1684243827.454169"", ""content"": ""hmmm possible looks like there is still some interlocks do you recollect which test failed in morning?""}, {""user"": ""ppant"", ""timestamp"": ""1684243918.899589"", ""content"": ""`LinkVendorCustomerTest.testGetCustomersWithoutBeingOrgAdmin`""}, {""user"": ""aganivada"", ""timestamp"": ""1684243951.394609"", ""content"": ""ok let me annotate with same_thread for this test and check if it fails again""}, {""user"": ""aganivada"", ""timestamp"": ""1684245242.920919"", ""content"": ""<@U0431DZTPJM> ran 3 builds haven't observed any failures so far""}, {""user"": ""ppant"", ""timestamp"": ""1684245459.193709"", ""content"": ""Thanks! So it must have been because that test class was getting executed in multiple threads""}, {""user"": ""aganivada"", ""timestamp"": ""1684245526.432079"", ""content"": ""Yeah we can revisit why the test fails when we run in multiple threads""}]" "1685418540.326999 ","[{""user"": ""aganivada"", ""timestamp"": ""1685418540.326999"", ""content"": ""<@U03KLHDKL1H> looks like snowflake supports sts are we using the same approach? this route should be fine we don't want to add a user and generate creds for the user that will be used elsewhere in snowflake? cc: <@U04JT69T00K>""}, {""user"": ""askumar"", ""timestamp"": ""1685418938.414339"", ""content"": ""sure <@U02BV2DGUKC> will review and make changes.\nThis is not the approach we followed.""}, {""user"": ""aganivada"", ""timestamp"": ""1685419219.432659"", ""content"": ""ok can we test this with int? please add a task <@U03KLHDKL1H>""}, {""user"": ""askumar"", ""timestamp"": ""1685419251.142489"", ""content"": ""yes""}, {""user"": ""askumar"", ""timestamp"": ""1685512583.195659"", ""content"": ""<@U02BV2DGUKC>\nwe able to configure the STS option 1 for the snowflake in INT.\nThe file load works and data is populated.\n\nShould we configure it for prod now ?\ncc <@U04JT69T00K>""}, {""user"": ""aganivada"", ""timestamp"": ""1685513013.341399"", ""content"": ""sure <@U03KLHDKL1H>\n\n<@U026PMDB1ND> FYI we are planning to integrate snowflake with prod data, integration will use STS model similar to gitlab""}, {""user"": ""askumar"", ""timestamp"": ""1685539431.122639"", ""content"": ""<@U026PMDB1ND>\nThe Prod account set up is complete for snowflake.\nNow data from both INT and PROD S3 env bucket will be available in snowflake\nand we can filter based on the Env filter that we have added to dashboard.\n\nThankyou <@U04JT69T00K> <@U02BV2DGUKC> for helping out with AWS production work.""}]" "1683658391.433229 ","[{""user"": ""pjha"", ""timestamp"": ""1683658391.433229"", ""content"": ""** \n*Steps to configure Group runner for Axiamatic Main :*\n1. *Get the Registration token :* Group(Axiamatic Main) *>* CI/CD *>* Runners > \u22ee (three dots on right)\n2. *Register the Group Runner :* Login to EC2(GL runner) instance *>* Register the gitlab runner with the Registration token *>* Restart the GL runner process .\n3. *Enable Run Untagged Jobs :* Once the runner is registered edit runner on Group > CI/CD > Runners page, here we need to edit the runner and enable _Run untagged jobs._\n4. *Disable the shared runner(project) :* {Project} *>* Settings *>* CI/CD *>* Runners, we need to disable shared runner for the jobs to be picked by the group runner.\n5. *Create weebhook* *:* Create weebhook on {Project} *>* Settings *>* Weebhook. \nRefer : \n\n<@U026PMDB1ND> Since I don't have permission to perform any action for *Axiamatic Main*(Group) therefore need to help for step 1 and 3\ncc: <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1683697254.149139"", ""content"": ""<@U04JT69T00K> I have permissions now, we can try updating and test post lunch""}, {""user"": ""pjha"", ""timestamp"": ""1683697274.028279"", ""content"": ""sure""}]" "1676263919.813159 ","[{""user"": ""aganivada"", ""timestamp"": ""1676263919.813159"", ""content"": ""<@U0431DZTPJM> moving the notification issue thread to platform channel:\n\n> `failed to send smtp unknown error` \n> *anil* [10:14 AM]\n> hmmm did we find reason for this?\n> *Prabhu Pant* [10:19 AM]\nI looked into it, it was because of this `com.sun.mail.smtp.SMTPAddressFailedException: 501 Invalid RCPT TO address provided`""}, {""user"": ""aganivada"", ""timestamp"": ""1676263935.662549"", ""content"": ""do we know which usecase trigerred this?""}, {""user"": ""ppant"", ""timestamp"": ""1676264840.444049"", ""content"": ""Could not find anything as such. The email id in the logs was ``. ""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1676275016.137869"", ""content"": ""Looks like the issue is bcoz of invalid email address""}, {""user"": ""aganivada"", ""timestamp"": ""1676275975.560019"", ""content"": ""user's email is wrong in DB, <@U02HCMTQU3W> /<@U02GC8SE18V> do we have email validation enabled during PVT creation or during bulk user imports? checking how we got into this state""}, {""user"": ""rvaidya"", ""timestamp"": ""1676276101.945089"", ""content"": ""<@U02BV2DGUKC> if the PVT was created before the integration was enabled, we just do domain validation. Otherwise we do user validation before adding it in the team.""}, {""user"": ""aganivada"", ""timestamp"": ""1676276400.527809"", ""content"": ""got it <@U02GC8SE18V>, but do we check if email matches general email patterns also? reason is this user's email might have passed domain validation though email id has invalid chars""}, {""user"": ""rvaidya"", ""timestamp"": ""1676277777.747569"", ""content"": ""Yes <@U02BV2DGUKC> we dont do any email validation while adding user in the team.\n\nThe only place where UI explicitly calls for bulk validation is with flow for importing user via csv.""}, {""user"": ""aganivada"", ""timestamp"": ""1676278075.861309"", ""content"": ""got it thank you <@U02GC8SE18V>, will add a UI ticket for email validation.\n\n<@U033PPLNFRU> can we update the user email directly from backend? or should we ask customer to remove and add a fresh user? user () may not have received any email's sent from axm""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1676278534.902299"", ""content"": ""<@U02BV2DGUKC> - Is this web only email tenant - if tenant has slack/teams enabled - the same user might be facing problem there too ?""}, {""user"": ""aganivada"", ""timestamp"": ""1676279889.037489"", ""content"": ""<@U0336QZAF98> looks like we sent survey through webapp so assuming this is a webapp tenant .\n\n<@U02GC8SE18V> /<@U02HCMTQU3W> if token-manager get enabled-services returns empty does it mean it is a webapp tenant?""}, {""user"": ""rvaidya"", ""timestamp"": ""1676280327.425789"", ""content"": ""Yes <@U02BV2DGUKC>""}]" "1684755000.970569 ","[{""user"": ""aganivada"", ""timestamp"": ""1684755000.970569"", ""content"": ""<@U03KLHDKL1H> looks like createProductOmnibus might have been flagged as read-only this is causing errors in int after enabling routing . for now I disabled read-only routing in core-data. we need to double check on the transactions flagged as read-only.""}, {""user"": ""aganivada"", ""timestamp"": ""1684755024.240069"", ""content"": ""<@U028EDANJM9> can we check now?""}, {""user"": ""aganivada"", ""timestamp"": ""1684755597.520999"", ""content"": ""<@U03KLHDKL1H> looks like the issue is this method is first trying to check if a product exists which is causing the entire transaction to be considered as read-only. Shouldn't library only consider the transaction marked as read-only as read-only?""}, {""user"": ""aganivada"", ""timestamp"": ""1684755713.398219"", ""content"": ""added for this""}, {""user"": ""gshenoy"", ""timestamp"": ""1684755850.263039"", ""content"": ""Works now <@U02BV2DGUKC>, Thank you\nCc <@U0281D3GWHL> <@U02HCMTQU3W>""}, {""user"": ""askumar"", ""timestamp"": ""1684755929.562759"", ""content"": ""<@U02BV2DGUKC> that's correct,\n If there are two transactions in a thread, first read then write, in that case we should not annotate it with read only.\nLooks like annotation was not setup correctly for this method or this method is being called from 2 places that might have caused the conflict at time of code changes.""}, {""user"": ""aganivada"", ""timestamp"": ""1684756068.364749"", ""content"": ""<@U03KLHDKL1H> I dont see read-only flag on controller may be some issue with transaction manager considering it as read-only since the first transaction is a read""}, {""user"": ""askumar"", ""timestamp"": ""1684756242.357599"", ""content"": ""<@U02BV2DGUKC>\nAlso there are no read only annotation on delegates, this looks like internal AWS library issue only.\n\nCould you please add the traceId here, that I can use later.""}, {""user"": ""gshenoy"", ""timestamp"": ""1684756268.935129"", ""content"": ""<@U03KLHDKL1H> 00000000000000003c03a063612f66c4""}, {""user"": ""aganivada"", ""timestamp"": ""1684756299.542249"", ""content"": ""<@U03KLHDKL1H> ""}, {""user"": ""askumar"", ""timestamp"": ""1684756340.847089"", ""content"": ""Thanks""}, {""user"": ""aganivada"", ""timestamp"": ""1684756376.306959"", ""content"": ""I remember we fixed this but not sure if something changed since""}, {""user"": ""askumar"", ""timestamp"": ""1684756455.256379"", ""content"": ""No <@U02BV2DGUKC> as we can see there is no annotation at all in this thread, so it must be something with lib only.""}, {""user"": ""aganivada"", ""timestamp"": ""1684756598.791519"", ""content"": ""<@U03KLHDKL1H> didn't we have an issue with library that it was considering entire transaction as read-only if the first transaction was a get call? I thought we fixed this through transaction manager to only give priority to our annotation instead of deriving by itself""}, {""user"": ""aganivada"", ""timestamp"": ""1684756657.260039"", ""content"": ""if lib can't fix it I guess we'd have to check by setting readonly to false forcefully?""}, {""user"": ""askumar"", ""timestamp"": ""1684756884.031659"", ""content"": ""Yes <@U02BV2DGUKC> the behavior is this only, where the transaction manager gives priority to readonly flag, only when annotation is there.\nIt does not derive by itself, else all the write APIs should have started failing randomly, because we have not annotated them.\n\nIf we can't figure issue in library then we can take the route of setting it forcefully.""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1684756915.704879"", ""content"": ""Works now, thanks <@U02BV2DGUKC> <@U028EDANJM9> and <@U03KLHDKL1H>""}]" "1672690702.300859 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1672690702.300859"", ""content"": ""<@U02BV2DGUKC> - added an epic to track flags that could/should be retired. I see that there are 3 platform flags which have targeting set to off but are returning false (all from 0.9.4). Do they correspond to features that are on the back-burner or something?\n\nLogged a ticket to retire the flag which is returning 'true' now.""}, {""user"": ""aganivada"", ""timestamp"": ""1672726816.321799"", ""content"": ""Yes <@U026PMDB1ND> security-settings and SSO related ones are on back-burner. Rbac mandate we don't need it anymore, initially we thought we could FF rbac but later we figured it add more complexity than solving the purpose so we discarded it. Will add a task to cleanup tenancy FF's in this sprint""}]" "1680029086.585659 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1680029086.585659"", ""content"": ""<@U04JT69T00K> - do you think we can set something up like this easily:\n\n\u2022 build docker image locally and tag it with the user or Jira name\n \u25e6 we use application/tools/ci-build.sh + some commands in gitlab-ci to build an image today\n \u25e6 We have a application/tools/local-build.sh as well, but I don't think anyone has used this after the first couple of months of Axmiatic.\n\u2022 upload this image to an ECR repo in INT (that we'd setup)\n\u2022 update CDK to use this repo location and deploy?\nWhile this would be very useful once we have the multiple instances story in place, it would be useful even now since we can cut out the whole GitLab build process (which takes a lot of time if we have to build a library and then a service followed by a deployment)\n\nThis could be a huge time saver in the cases where a single dev works on a specific service.""}, {""user"": ""pjha"", ""timestamp"": ""1680032275.398649"", ""content"": ""I think it shouldn't be too difficult to get this done.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680037583.610499"", ""content"": ""Nice, can you pick this up in the next sprint?""}, {""user"": ""pjha"", ""timestamp"": ""1680062934.223399"", ""content"": ""sure""}, {""user"": ""pjha"", ""timestamp"": ""1680534649.628119"", ""content"": ""<@U026PMDB1ND> I wanted to reach out and ask for your opinion, here we have 2 approach :\n1. Creating ECR Repo, building image locally and pushing it to the repositories can be done before the deployment and the required value will be passed to the CDK code.\n2. We can provide the Dockerfile path to the CDK code which will build image locally and push it to the ECR repo(created through cdk ) and we can use the image for the ECS task .\n For this we can use 3rd-party construct library(cdk_ecr_deployment). downside is that on destroying the stacks won't destroy the ECR repo. Refer: \ncc: <@U02BV2DGUKC>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680535058.324099"", ""content"": ""<@U04JT69T00K> (1) is good, IMO""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680535194.395979"", ""content"": ""We already have some script that cleans up images. If we include \u2018local\u2019 in the image name of these images, we can update the script to kill things older than a week or two.""}, {""user"": ""pjha"", ""timestamp"": ""1680535262.554599"", ""content"": ""<@U026PMDB1ND> got it\nThanks""}]" "1679001893.104389 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1679001893.104389"", ""content"": ""<@U03DHUAJVMK> <@U03NZ7Z52S2> -\nIn prod:\n\n```postgresdbproduction=> select count(*) from publisher where tenant_id=1932;\n count\n-------\n 4084\n(1 row)```\nWhich test keeps creating these entities and not deleting them afterwards?""}, {""user"": ""svummidi"", ""timestamp"": ""1679003478.140469"", ""content"": ""Here is one old discussion related to this topic. <@U02HQ78V9A5> suspecting that we are not doing cleanup after tests.\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679031083.769629"", ""content"": ""I updated the test code to replace the long UUID with shortuuid and the test name for some of the entities. We did that to track down tenants that we leave behind and had success.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679031124.814309"", ""content"": ""Once this code gets merged we should be able to figure this out easily. fyi <@U03NZ7Z52S2>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679031345.156299"", ""content"": """"}, {""user"": ""bganganna"", ""timestamp"": ""1679040883.683699"", ""content"": ""<@U026PMDB1ND> I found some suites where we are creating the publishers and there is no tear down , but those does not have sanity or regression tag its mostly post-deploy, and where we have added the sanity/regression tags i could see the teardown for publishers.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679066913.446149"", ""content"": ""<@U03NZ7Z52S2> - can you publish that list and/or fix the issues?""}, {""user"": ""bganganna"", ""timestamp"": ""1679318112.381659"", ""content"": ""<@U026PMDB1ND> My bad i missed two keywords :Create Product Instance For PVT , Create Product Instance For PVT AND GET PRODUCT ID. This is called in most of the app test-cases.\napps_common.Product Value Team Cleanup called during the teardown in most of the test cases . I dont think we are deleting the publishers. Will add that as part of this teardown.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679328310.203339"", ""content"": ""<@U03NZ7Z52S2> - let's try to get this done soon""}]" "1685441591.957109 ","[{""user"": ""ppant"", ""timestamp"": ""1685441591.957109"", ""content"": "" <@U02BV2DGUKC> Notification-service\u2019s tests are not compatible with latest versions of axm-commons. Getting this unsatisfied dependency exception in many test classes after updating axm-commons from 0.0.8 to 0.0.17 in its pom\n```Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.springframework.boot.autoconfigure.security.oauth2.client.OAuth2ClientProperties' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}\n\tat org.springframework.beans.factory.support.DefaultListableBeanFactory.raiseNoMatchingBeanFound(DefaultListableBeanFactory.java:1801)\n\tat org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1357)```""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1685442227.321619"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1685442604.225009"", ""content"": ""<@U0336QZAF98> is this tenancy change related?""}, {""user"": ""aganivada"", ""timestamp"": ""1685442632.584349"", ""content"": ""may be we need to add mock s2swebclient?""}, {""user"": ""ppant"", ""timestamp"": ""1685442633.250049"", ""content"": ""Tried adding the mock s2sWebClient but still not working. Same issue pops up""}, {""user"": ""aganivada"", ""timestamp"": ""1685442676.298339"", ""content"": ""<@U0431DZTPJM> can you compare application-test.yml with other services?""}, {""user"": ""ppant"", ""timestamp"": ""1685442774.643669"", ""content"": ""Also added `@ContextConfiguration(classes = {TestConfig.class})` at the top of test classes which were failing, still nothing""}, {""user"": ""ppant"", ""timestamp"": ""1685442792.315279"", ""content"": ""<@U02BV2DGUKC> Compared test yml of notif with tenancy, nothing as such""}, {""user"": ""aganivada"", ""timestamp"": ""1685442887.353789"", ""content"": ""<@U0431DZTPJM> does mvn spring-boot:run work? just wondering if this is a test only issue""}, {""user"": ""ppant"", ""timestamp"": ""1685442933.575129"", ""content"": ""Build works fine, only tests are failing""}, {""user"": ""aganivada"", ""timestamp"": ""1685442975.534229"", ""content"": ""ok""}, {""user"": ""aganivada"", ""timestamp"": ""1685443383.207959"", ""content"": ""```spring:\n security:\n oauth2:\n client:\n registration:\n s2s:\n client-id: ${S2S_CLIENT_ID:-}\n client-secret: ${S2S_CLIENT_SECRET:-}\n authorization-grant-type: client_credentials\n provider:\n s2s:\n token-uri: ${S2S_TOKEN_URL:-}```\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685461824.313779"", ""content"": ""<@U0431DZTPJM> - have we resolved this?""}, {""user"": ""ppant"", ""timestamp"": ""1685461864.391719"", ""content"": ""Yes <@U026PMDB1ND>. Adding the above configs and mock bean fixed this. Have opened a MR for this as well ""}]" "1694619849.085899 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1694619849.085899"", ""content"": ""<@U02BV2DGUKC> - did you modify the story points question at the end of the retro or did a bug creep in?""}, {""user"": ""aganivada"", ""timestamp"": ""1694619937.701399"", ""content"": ""seems like a bug <@U026PMDB1ND> ""}]" "1680220860.347599 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1680220860.347599"", ""content"": ""<@U040RCBPBEC> - there's an exception on the ScoreProvider side for EA Tenant. This is the root exception:\n\n```Caused by: org.apache.calcite.avatica.AvaticaSqlException: Error -1 (00000) : Remote driver error: QueryInterruptedException: java.util.concurrent.ExecutionException: QueryInterruptedException{msg=null, code=Unsupported operation, class=java.lang.UnsupportedOperationException, host=ip-10-51-196-110.us-west-2.compute.internal:8283} -> RuntimeException: java.util.concurrent.ExecutionException: QueryInterruptedException{msg=null, code=Unsupported operation, class=java.lang.UnsupportedOperationException, host=ip-10-51-196-110.us-west-2.compute.internal:8283} -> ExecutionException: QueryInterruptedException{msg=null, code=Unsupported operation, class=java.lang.UnsupportedOperationException, host=ip-10-51-196-110.us-west-2.compute.internal:8283} -> QueryInterruptedException: (null exception message)\n\tat org.apache.calcite.avatica.Helper.createException(Helper.java:54)\n\tat org.apache.calcite.avatica.Helper.createException(Helper.java:41)\n\tat org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:520)\n\tat org.apache.calcite.avatica.AvaticaPreparedStatement.execute(AvaticaPreparedStatement.java:231)\n\tat com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)\n\tat com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)\n\tat org.jooq.tools.jdbc.DefaultPreparedStatement.execute(DefaultPreparedStatement.java:219)\n\tat org.jooq.impl.Tools.executeStatementAndGetFirstResultSet(Tools.java:4330)\n\tat org.jooq.impl.AbstractResultQuery.execute(AbstractResultQuery.java:230)\n\tat org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:340)\n\t... 65 more```\n\n\n\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680220881.997369"", ""content"": ""fyi <@U033PPLNFRU>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680220912.049659"", ""content"": ""<@U033PPLNFRU> , are you accessing their tenant?""}, {""user"": ""svummidi"", ""timestamp"": ""1680221007.756429"", ""content"": ""<@U026PMDB1ND> <@U033PPLNFRU> - Currently there is a problem due to EA data re-indexing. Discussing with Druid support team. We may see issues in UI while trying to access the page.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680221024.560669"", ""content"": ""Ok""}, {""user"": ""anair"", ""timestamp"": ""1680222757.465879"", ""content"": ""ack <@U040RCBPBEC> <@U026PMDB1ND>""}]" "1681799669.540539 ","[{""user"": ""nsrivastava"", ""timestamp"": ""1681799669.540539"", ""content"": "" I am getting `docker` missing error in one of the pipelines here , has anyone seen this before or aware of the probable cause? this is the MR ""}, {""user"": ""aganivada"", ""timestamp"": ""1681799728.335499"", ""content"": ""<@U03RQDE3QUS> this is because GL runner changes have not yet been merged to develop branch""}, {""user"": ""aganivada"", ""timestamp"": ""1681799743.882549"", ""content"": ""redirecting pieline to regular shared runner""}, {""user"": ""nsrivastava"", ""timestamp"": ""1681799776.770319"", ""content"": ""got it <@U02BV2DGUKC>, thanks.""}, {""user"": ""aganivada"", ""timestamp"": ""1681799836.360529"", ""content"": ""np <@U03RQDE3QUS>, please trigger the pipeline again, this is being worked upon here once all tests related issues are resolved we can merge the GL-runner changes to develop""}, {""user"": ""nsrivastava"", ""timestamp"": ""1681800718.361669"", ""content"": ""thanks <@U02BV2DGUKC> job succeeded with this.""}]" "1680740083.605709 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1680740083.605709"", ""content"": ""<@U02BV2DGUKC> - whenever I login to LD they show a notice that their SDK is reaching EOL and we should upgrade. I'll do that in the next couple of sprints. Looks like they made some non-trivial changes""}, {""user"": ""aganivada"", ""timestamp"": ""1680755282.689089"", ""content"": ""sure <@U026PMDB1ND> , added for this""}]" "1691732236.853499 ","[{""user"": ""akasim"", ""timestamp"": ""1691732236.853499"", ""content"": ""\nGetting the following error in MB develop branch build. Can some one help?\ncc: <@U02BV2DGUKC>\n```2146 [ERROR] [ERROR] Some problems were encountered while processing the POMs:\n38[FATAL] Non-resolvable parent POM for com.axm.collaboration:message-broker-parent:0.0.20-SNAPSHOT: Could not transfer artifact com.axm.platform:axm-parent:pom:0.0.22 from/to gitlab-maven (): authentication failed for , status: 401 Unauthorized and 'parent.relativePath' points at no local POM @ line 10, column 13```""}, {""user"": ""aganivada"", ""timestamp"": ""1691738016.399199"", ""content"": ""<@U02HQ78V9A5> this is happening locally or in gitlab?""}, {""user"": ""akasim"", ""timestamp"": ""1691738033.107279"", ""content"": ""gitlab <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1691738054.343749"", ""content"": ""ok taking a look""}, {""user"": ""akasim"", ""timestamp"": ""1691738066.594039"", ""content"": ""Thanks""}, {""user"": ""aganivada"", ""timestamp"": ""1691738702.912889"", ""content"": ""<@U02HQ78V9A5> not sure why this is happening all of a sudden but trying this change ""}, {""user"": ""akasim"", ""timestamp"": ""1691738782.485339"", ""content"": ""Ok sure <@U02BV2DGUKC>. It's quite surprising that why its building successfully in local.""}, {""user"": ""aganivada"", ""timestamp"": ""1691739023.357709"", ""content"": ""<@U02HQ78V9A5> build is working now.\n\non root-cause this was the response 2 months back, there might be some race condition due to this command not sure""}, {""user"": ""akasim"", ""timestamp"": ""1691739080.427719"", ""content"": ""Great :+1:\nThanks for the help <@U02BV2DGUKC>""}]" "1677754540.935489 ","[{""user"": ""amishra"", ""timestamp"": ""1677754540.935489"", ""content"": ""<@U02BV2DGUKC> I'm getting \""Could not find products by name. search-registry-product-error|809|Find products matching filter {name=ilike='tes'} failed\""\n\n*env* - INT\n*api* - /product/search?pageSize=1000&productName=test\n*trace-id:* 64008063274b9c4b0b722a2bc5febc13""}, {""user"": ""aganivada"", ""timestamp"": ""1677754866.965709"", ""content"": ""<@U04ARFN3WQP> this user is a orgadmin error doesnt looks like permission or RBAC issue. is this reproducible continously? coz in integration we've been bouncing core to test some changes""}, {""user"": ""aganivada"", ""timestamp"": ""1677770041.378129"", ""content"": ""<@U04ARFN3WQP> did this resolve?""}, {""user"": ""amishra"", ""timestamp"": ""1677771855.151119"", ""content"": ""No, <@U02BV2DGUKC> still experiencing the same error on INT, its working on Stage""}, {""user"": ""aganivada"", ""timestamp"": ""1677772472.093669"", ""content"": ""<@U04ARFN3WQP> can you try some other text? I am wondering if there is some stale data in int that is causing the issue""}, {""user"": ""aganivada"", ""timestamp"": ""1677772507.876479"", ""content"": ""there might be some record with wrong productType\n\n> \t java.lang.IllegalArgumentException\n> \tat java.base/java.util.Optional.orElseThrow(Optional.java:408)\n> \tat com.axm.coredata.commons.enums.ProductType.of(ProductType.java:27)\n> \tat com.axm.core.services.coredata.mapper.CoreDataMapper.mapProductType(CoreDataMapper.java:367)""}, {""user"": ""aganivada"", ""timestamp"": ""1677773681.822269"", ""content"": ""<@U028EDANJM9> it seems some products got inserted with names instead of id's for producttype in integration db which are creating issue while transforming data does this require a cleanup?""}, {""user"": ""gshenoy"", ""timestamp"": ""1677773941.950449"", ""content"": ""<@U02BV2DGUKC> most likely these were inserted using API instead of cloud config, as I don\u2019t find the product name `product-sp-test-vendor1` in the config files. Will clean this up.\nCc <@U02T4E0BVA8>""}, {""user"": ""aganivada"", ""timestamp"": ""1677774161.767509"", ""content"": ""ok thank you <@U028EDANJM9>\n\n<@U04ARFN3WQP> this issue is due to stale data associated with the text in integration env we can ignore it for now and try other strings""}, {""user"": ""amishra"", ""timestamp"": ""1677774657.748979"", ""content"": ""Thanks <@U02BV2DGUKC> for clarification, other strings work""}, {""user"": ""gshenoy"", ""timestamp"": ""1677776108.255229"", ""content"": ""Cleaned up the wrong values <@U04ARFN3WQP> <@U02BV2DGUKC>""}]" "1689256591.686259 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1689256591.686259"", ""content"": ""<@U02BV2DGUKC> - Is pulse manager down in INT ?\n""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689256749.833779"", ""content"": ""Its up now""}, {""user"": ""aganivada"", ""timestamp"": ""1689259823.893489"", ""content"": ""hmmm probably restarting at the time?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689260844.900939"", ""content"": ""The way restart is managed by ECS we shouldn't have a gap in availability.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689260853.434469"", ""content"": ""Even though there's only 1 instance""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689260947.909299"", ""content"": ""<@U0336QZAF98> do you remember the absolute time when this was the case?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689261149.675219"", ""content"": ""It was silent for 2 minutes and indeed there was restart""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689261154.936219"", ""content"": ""```From 7.20 PM to & 7.30 PM between\n2d90365f-0f6c-40ff-a8c6-55089b644476\n2023-07-13 19:27:56 +0530\nservice pulse-manager has reached a steady state.\n41f87c0d-e834-4a5e-903b-206165c5e2e3\n2023-07-13 19:25:38 +0530\nservice pulse-manager registered 1 targets in target-group pulse-albpu-17IV0KOO7XX8S\nf199d464-c5c9-4f86-a5c3-8ab3769ebf76\n2023-07-13 19:24:50 +0530\nservice pulse-manager has started 1 tasks: task 1eff6f9f293e4e4585a9048838db711c.\n4eb028d0-a11f-42e0-8033-93a386f382a4\n2023-07-13 19:24:49 +0530\n(service pulse-manager, taskSet ecs-svc/9813861328219907657) has begun draining connections on 1 tasks.\n7cb776aa-1401-45c2-8582-a665688b2ede\n2023-07-13 19:24:49 +0530```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689261196.173809"", ""content"": ""<@U0336QZAF98> - where this is output from?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689261224.085429"", ""content"": ""This is from Events tab in - old ECS Cluser view""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689261301.733329"", ""content"": """"}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689261356.083069"", ""content"": """"}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689261359.976999"", ""content"": ""There was a huge cpu utilization""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689261361.889629"", ""content"": ""Can you see what happened to the task 448*""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689261443.915159"", ""content"": ""Entire history of stopped tasks - i am not able to see here - Typically it comes here""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689261450.654169"", ""content"": ""is empty""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689261587.892199"", ""content"": ""I think the memory is keep on increasing - there is a leak or something we are doing here""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689261668.724259"", ""content"": "" - Have you guys recently monitored pulse manager memory usage - it keeps on increasing over a period of time - FYI""}, {""user"": ""mnirmal"", ""timestamp"": ""1689261758.771049"", ""content"": ""I took a dump sometime back <@U0336QZAF98> when I noticed this increasing, but couldn't find anything.""}, {""user"": ""mnirmal"", ""timestamp"": ""1689261778.915299"", ""content"": ""I was planning on taking one after a few days, but lost track of it.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689262002.910219"", ""content"": ""could you add a backlog for the same <@U02SF36PVKL>""}, {""user"": ""aganivada"", ""timestamp"": ""1689262048.073109"", ""content"": ""<@U02SF36PVKL> will add a cloudwatch alert on this node on memory so we can take a memory dump""}, {""user"": ""mnirmal"", ""timestamp"": ""1689262082.696049"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1689266187.691229"", ""content"": ""<@U02SF36PVKL> added the alert and tagged notification to <#C03KDBFCMM2|on-call> channel\n\n?\n\nif possible please take a memory dump now so we can compare once memory crosses threshold.""}]" "1683532298.230219 ","[{""user"": ""sjain"", ""timestamp"": ""1683532298.230219"", ""content"": "",\nMy builds for sentiment-collector are failing while publishing docker image with below error.\n\n`aws: error: argument --role-arn: expected one argument`\nJob url : \nCan anyone tell me what needs to be fixed . ?""}, {""user"": ""ppant"", ""timestamp"": ""1683532350.374509"", ""content"": ""<@U02TVMF3CR4> You need to mark your branch as protected. Its unable to access some gitlab variable""}, {""user"": ""aganivada"", ""timestamp"": ""1683532511.847419"", ""content"": ""<@U02TVMF3CR4> by default only develop, main and HF branches are marked as protected other branches don't have necessary permission to publish. If you want to publish the image as Prabhu pointed out we will have to mark the branch protected. Let me do that for this branch""}, {""user"": ""aganivada"", ""timestamp"": ""1683532655.493019"", ""content"": ""done and re-triggered ""}, {""user"": ""sjain"", ""timestamp"": ""1683532724.681149"", ""content"": ""My bad, the third pipeline build used to be \u201cdeploy build\u201d , didn\u2019t realised the ci/cd pipeline changed & now only has 2 builds.""}, {""user"": ""sjain"", ""timestamp"": ""1683532732.321769"", ""content"": ""thanks <@U02BV2DGUKC>""}]" "1691776448.652279 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1691776448.652279"", ""content"": ""<@U04JT69T00K> <@U02BV2DGUKC> - might be useful to to run askai build on our EC2 runner...""}, {""user"": ""aganivada"", ""timestamp"": ""1692016116.224059"", ""content"": "" jfyi attempting to move askai to GL-runner""}, {""user"": ""aganivada"", ""timestamp"": ""1692018672.036479"", ""content"": ""reverted back\n\n<@U04JT69T00K> once you are back can you take a look at this? was trying to use project based runner but the job is not getting picked up by GL-runner, webhook is also working.""}]" "1687950531.285219 ","[{""user"": ""ppant"", ""timestamp"": ""1687950531.285219"", ""content"": "" Seems like I have made too many commits for multi-instance changes that Gitlab\u2019s runner has blocked tenancy""}, {""user"": ""aganivada"", ""timestamp"": ""1687951040.417999"", ""content"": ""Hmmm previous failures?""}, {""user"": ""aganivada"", ""timestamp"": ""1687951069.213999"", ""content"": ""Should we switch to shared runner?""}, {""user"": ""ppant"", ""timestamp"": ""1687951072.505379"", ""content"": ""Yes, there were 6-7 failures. I am running core-data\u2019s multi instance branch pipeline again and then will re-run tenancy""}, {""user"": ""aganivada"", ""timestamp"": ""1687951129.976419"", ""content"": ""Ok""}, {""user"": ""aganivada"", ""timestamp"": ""1687951156.129779"", ""content"": ""Is this in branch or develop?""}, {""user"": ""ppant"", ""timestamp"": ""1687951182.322649"", ""content"": ""Its my MR\u2019s branch ""}, {""user"": ""ppant"", ""timestamp"": ""1687951635.980019"", ""content"": ""It is passing now ""}, {""user"": ""ppant"", ""timestamp"": ""1687951693.026599"", ""content"": ""The problem was that this MR is dependent on core-data\u2019 2052-SNAPSHOT version which somehow got deleted from package registry after 2 days. Rebuilt it again to and re-ran the pipeline""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687972552.651069"", ""content"": ""<@U0431DZTPJM> - you sure it was built recently?\n\nThis is the log from the artifact reaper job:\n```2023-06-28 05:03:23,142 - root - INFO - Deletable: com/axm/platform/services/core-data-service 0.0.16-PLAT-2052-SNAPSHOT 31089821 14738951 2023-05-19T14:41:13.563Z\n\n2023-06-28 05:03:50,686 - root - INFO - Deleted com/axm/platform/services/core-data-base 0.0.16-PLAT-2052-SNAPSHOT 31089821 14738941 2023-05-19T14:40:52.774Z status_code: 204```\nIt sees the timestamp as *2023-05-19T14:40:52.774Z*""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687972580.558379"", ""content"": ""We delete snapshots older than 21 days""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687973893.226489"", ""content"": ""<@U04JT69T00K> - did we get an alert when Prabhu ran into this?""}, {""user"": ""ppant"", ""timestamp"": ""1687975140.330809"", ""content"": ""Hmm. Then I think my memory is serving me wrong :thinking_face:""}]" "1685030656.536229 ","[{""user"": ""aganivada"", ""timestamp"": ""1685030656.536229"", ""content"": ""<@U04JT69T00K> reposting question from <@U02HCMTQU3W> from zoom to slack\n\n> Can we find all the stacks that are deployed via multi-env?Via a prefix or a tag on the stack?""}, {""user"": ""pjha"", ""timestamp"": ""1685031006.131389"", ""content"": ""no, slack message doesn't specify the deployment version""}, {""user"": ""aganivada"", ""timestamp"": ""1685031041.998309"", ""content"": ""<@U04JT69T00K> can we add a tag for all multi-instance deployments in aws?""}, {""user"": ""pjha"", ""timestamp"": ""1685031184.516609"", ""content"": ""yes, we can add the tags to the resources created through multi-instance-deployment.""}]" "1691084550.001809 ","[{""user"": ""nsrivastava"", ""timestamp"": ""1691084550.001809"", ""content"": "" I am trying to publish a npm package onto gitlab registry, npm seems to have disabled overwriting the same version. I tried un-publishing but getting this error\n\n```npm ERR! Refusing to delete the last version of the package. It will block from republishing a new version for 24 hours.```\nthis is the job: \n\ndoes anyone know if there is a way to overwrite or do away with 24 hours timeline, or if we could get gitlab support on this. The one last option could be to append a guid in the snapshot versions but trying to avoid that. cc: ""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1691085465.497779"", ""content"": ""<@U03RQDE3QUS> - in python we explicitly delete via this route\n``` APP_VERSION=$(poetry version -s)\n JQ_FILTER=\"".[] | select( (.version |ascii_downcase == \\\""$APP_VERSION\\\"") or (.version |ascii_upcase == \\\""$APP_VERSION\\\"") )\""\n EXISTING_PACKAGE=$(gitlab --private-token ${CI_FULL_ACCESS_TOKEN} --fields id,version -o json project-package list --project-id $CI_PROJECT_ID --order-by version --sort desc --per-page 100 | jq -c \""$JQ_FILTER\"" | jq \"".id\"")\n echo \""existing package - {$EXISTING_PACKAGE}\""\n if [ ! -z \""${EXISTING_PACKAGE}\"" ]; then \n echo \""removing existing package=${EXISTING_PACKAGE} and version=${APP_VERSION}\""\n gitlab --private-token ${CI_FULL_ACCESS_TOKEN} project-package delete --project-id $CI_PROJECT_ID --id $EXISTING_PACKAGE\n else\n echo \""unable to find existing package\""\n fi ```""}, {""user"": ""nsrivastava"", ""timestamp"": ""1691093647.292499"", ""content"": ""interesting, I was thinking we shouldn\u2019t be the first facing this issue on gitlab and hoping there would be some config to adjust this behavior on gitlab registry, google however din\u2019t produce much results, unpublish was meant for this but has a ~hard limit for 24 hours.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1691097302.569169"", ""content"": ""That's a reasonable guess. I ~can~ opened a case with them. Let's see what they say.""}, {""user"": ""nsrivastava"", ""timestamp"": ""1691123737.539679"", ""content"": ""Thanks <@U026PMDB1ND>""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1691131926.493629"", ""content"": ""<@U03RQDE3QUS> - your project runs in a docker container right - meaning you will be creating docker image - do we need to publish to npm ?""}, {""user"": ""nsrivastava"", ""timestamp"": ""1691132030.937499"", ""content"": ""<@U0336QZAF98> this is not regarding the project, teams-integrator need to invoke token manager APIs, hence was publish TM\u2019s spec as node module to registry.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1691132098.522609"", ""content"": ""ohh got it""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1691787544.417029"", ""content"": "" <@U03RQDE3QUS> <@U0336QZAF98> :\n\nThis is the response we received from GL:\n\nAs per this : a uniq index is added to the to the for npm packages to prevent having more then one npm package with the same `name`, `version`and `project_id`.\n\nOur product team is working on implementing the feature, which allows users to enable/disable duplicate packages:\n\n\u2022 *Epic*: **\nAs of now, duplicates are allowed for `maven`, `generic`, `helm`, `nuget` and `conan` package types. `npm` package type is not yet supported - we have reviewed your feedback and it seems that a feature request like this was already proposed, here's a link to it:\n\n\u2022 *Issue*: **\nTo help prioritize the feature proposal, we would like to add a comment with some information. If you prefer, you can add this information directly to the issue yourself, and we encourage you to upvote, and follow the issues for updates on its progress.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1691787948.070349"", ""content"": ""The ticket they asked us to upvote/comment on is totally dormant. Weird decision to do this for maven etc., but not for npm. The motivation is not very clear""}]" "1688390370.415719 ","[{""user"": ""aganivada"", ""timestamp"": ""1688390370.415719"", ""content"": ""<@U04JT69T00K> for mult-instance it seems we are not updating container name is this expected?""}, {""user"": ""pjha"", ""timestamp"": ""1688390684.543519"", ""content"": ""Code changes are not there to update the container name, I will make this change""}, {""user"": ""pjha"", ""timestamp"": ""1688394034.672749"", ""content"": ""Please review the commit\n""}, {""user"": ""aganivada"", ""timestamp"": ""1688394255.094319"", ""content"": ""LGTM <@U04JT69T00K> can you also post in merge-request channel.\n\nWhen are we planning to merge these changes to develop? Are there any known risks?""}, {""user"": ""pjha"", ""timestamp"": ""1688394526.852119"", ""content"": ""no such known risk, was waiting for SSM-Mapping changes, I think we can merge it to develop by this week.""}]" "1693812530.681339 ","[{""user"": ""aganivada"", ""timestamp"": ""1693812530.681339"", ""content"": ""<@U0281D3GWHL> we currently have a licensing issue with Perimeter 81, Our current account limit is set to 31 but we need 32 to allow access to Vishal (Not sure how this was working before device MFA got corrupted). I tried to add license from P81 UI but it is failing, Have raised a support ticket with them to look into it. Unfortunately, until this is resolved we cannot add <@U05PCBD9SHE> to P81. I have raised about severity of the issue with P81 will update as soon as I hear from them. cc: <@U03DHUAJVMK>""}, {""user"": ""aganivada"", ""timestamp"": ""1693813421.295259"", ""content"": ""<@U05PCBD9SHE> while we work with P81 please let us know if you need any details from DB/AWS. I guess you should still have access to int and stage UI.""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1693814593.194599"", ""content"": ""Thanks <@U02BV2DGUKC> is there any one we can de-provision or remove from P81 in the interim? Dejan is no longer with us, perhaps we can remove his account?""}, {""user"": ""U05PCBD9SHE"", ""timestamp"": ""1693816859.338099"", ""content"": ""<@U02BV2DGUKC> I have DB/aws access also I am able to access int and stage. So only issue is with VPN. ""}, {""user"": ""aganivada"", ""timestamp"": ""1693816902.480819"", ""content"": ""> Dejan is no longer with us, perhaps we can remove his account?\nsure thank you <@U0281D3GWHL>, ~I can replace Dejan's account with Vishal's~ Dejan did not have any account in P81 :disappointed:""}, {""user"": ""aganivada"", ""timestamp"": ""1693816949.485749"", ""content"": ""> I have DB/aws access also I am able to access int and stage.\n<@U04JT69T00K> did we disable P81 policies in int and stage?""}, {""user"": ""pjha"", ""timestamp"": ""1693817186.325529"", ""content"": ""<@U02BV2DGUKC> yes the policy is disabled in stage not in INT.""}, {""user"": ""pjha"", ""timestamp"": ""1693817283.779559"", ""content"": ""I will enable in stage.""}, {""user"": ""aganivada"", ""timestamp"": ""1693817615.197759"", ""content"": ""<@U04JT69T00K> please wait until few hours once we have <@U05PCBD9SHE>\u2019s access resolved we can enable policies.""}, {""user"": ""aganivada"", ""timestamp"": ""1693817637.376269"", ""content"": ""Also if policy is enabled in int how is Vishal able to access int?""}, {""user"": ""pjha"", ""timestamp"": ""1693817666.240219"", ""content"": ""Let me check""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1693817956.647199"", ""content"": ""What about Anandhi? Did we provision any account for her <@U02BV2DGUKC>?""}, {""user"": ""pjha"", ""timestamp"": ""1693818293.499919"", ""content"": ""<@U02BV2DGUKC> vishal is not able to access aws console or DB in INT without VPN.""}, {""user"": ""U05PCBD9SHE"", ""timestamp"": ""1693818372.088319"", ""content"": ""<@U02BV2DGUKC> sorry that was my mistake. I tried it now and I am not able to DB. Cc: <@U04JT69T00K> ""}, {""user"": ""aganivada"", ""timestamp"": ""1693819014.147239"", ""content"": ""> What about Anandhi? Did we provision any account for her <@U02BV2DGUKC>?\nno <@U0281D3GWHL> , we didn't create accounts for product as they were mostly accessing UI""}]" "1692894818.879419 ","[{""user"": ""aganivada"", ""timestamp"": ""1692894818.879419"", ""content"": ""<@U02GC8SE18V> <@U02HCMTQU3W> <@U03DHUAJVMK> jfyi added for 0.9.14 HF, this is a request to rename the vendor admin role in dashboard. Will be updating core-data-commons and dashboard-app pom as part of this change. cc: <@U033PPLNFRU>""}, {""user"": ""araman"", ""timestamp"": ""1692942568.531689"", ""content"": ""<@U02BV2DGUKC> Could you pls assign the ticket to me when its QA-Ready? Don't want to miss out a quick check""}, {""user"": ""aganivada"", ""timestamp"": ""1692942905.612119"", ""content"": ""Sure <@U03DHUAJVMK>""}, {""user"": ""aganivada"", ""timestamp"": ""1693193986.044769"", ""content"": ""<@U03DHUAJVMK> this change is deployed to stage now""}]" "1680777599.194859 ","[{""user"": ""askumar"", ""timestamp"": ""1680777599.194859"", ""content"": "" <@U02BV2DGUKC>\nDemo for AWS Aurora Read Write DB\n""}, {""user"": ""aganivada"", ""timestamp"": ""1680782988.076769"", ""content"": ""cool very nice thank you <@U03KLHDKL1H>, can you also voice over following during demo\n1. if someone attempts a write operation on a method flagged as read-only then the operation will fail\n2. application.yml changes where we mention read path and read-write path\n3. local config where there is only one type of instance\n""}, {""user"": ""askumar"", ""timestamp"": ""1680783092.565209"", ""content"": ""Sure <@U02BV2DGUKC>...would do that""}]" "1689140218.535949 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1689140218.535949"", ""content"": ""<@U04JT69T00K> - I understood minimum 8 acu charge to mean that at least we\u2019d spend 8* 0.015* 730 = 87.6 dollars per month.\n\nFor 5 clusters across 3 envs, we\u2019d be spending $438 per month at a minimum.\n\nI think we should go for it in INT first and see if we can reduce the max acu to save some money.""}, {""user"": ""pjha"", ""timestamp"": ""1689141792.757359"", ""content"": ""yes, we will be billed for minimum of 8 ACUs per hour.""}]" "1683007390.160799 ","[{""user"": ""askumar"", ""timestamp"": ""1683007390.160799"", ""content"": "" <@U026PMDB1ND>\nWe encountered an issue while migrating data via AWS Data Migration service.\n\nThe migration task is able to copy table data and primary keys, however it is not able to copy additional table meta data such as Foreign key constraints.\n\nTo overcome this we are currently thinking of below approach:\n1. Let the service create schema on new DB - this will add tables and all constraints\n2. Migrate the data in a way so that we can take care of foreign key relations. - This would mean multi step migration of tables, so that base tables are copied, then the dependent tables.\nIt may be tedious to figure out the relations, we will see if there is another safer option.\nPlease let us know if you have any suggestions on this.\ncc <@U02BV2DGUKC>""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1683007446.420569"", ""content"": ""<@U03KLHDKL1H> - did we look at pgdump - ""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1683007596.090059"", ""content"": """"}, {""user"": ""askumar"", ""timestamp"": ""1683007902.181249"", ""content"": ""thanks <@U0336QZAF98> for dump it takes longer time (15 min+), I tried long time back.\nWill take a relook.""}, {""user"": ""aganivada"", ""timestamp"": ""1683008621.980599"", ""content"": ""<@U03KLHDKL1H> since this migration is applicable only for int/stage 15+ mins should be ok in my opinion. Also this will be one time activity once we switch to new db we can delete the current instance""}, {""user"": ""askumar"", ""timestamp"": ""1683008709.244809"", ""content"": ""sure <@U02BV2DGUKC> relooking into this.""}, {""user"": ""aganivada"", ""timestamp"": ""1683008795.833939"", ""content"": ""We have to check how dump works in case of liquibase related tables or may be we can update them through the AWS Data Migration service.""}, {""user"": ""askumar"", ""timestamp"": ""1683008938.359479"", ""content"": ""Yes""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683048844.632239"", ""content"": ""<@U03KLHDKL1H> - do we have a final answer whether pg_dump route takes care of the metadata?""}, {""user"": ""askumar"", ""timestamp"": ""1683088735.556269"", ""content"": ""Hi <@U026PMDB1ND>\nYes, we could achieve migration of metadata as well.\n\n- AWS DMS for data migration\n- pg_dump for schema and constraints, sequence, sequence values migration\n\nThere are couple of manual steps, will try to close it in call.""}, {""user"": ""askumar"", ""timestamp"": ""1683112209.418599"", ""content"": ""Added AWS support ticket ;\n\n""}, {""user"": ""aganivada"", ""timestamp"": ""1683115637.199789"", ""content"": ""<@U03KLHDKL1H> can you include to mail chain""}, {""user"": ""askumar"", ""timestamp"": ""1683117371.536439"", ""content"": ""Sure""}, {""user"": ""askumar"", ""timestamp"": ""1683129734.100749"", ""content"": ""Update :\nThanks to <@U0336QZAF98>\nWe were able to migrate data with help of PgAdmin console\nConstraints and data both were migrated successfully.\n\nWithin 5 minutes we could take dump and do a restore.\ncc <@U02BV2DGUKC> <@U026PMDB1ND>""}, {""user"": ""askumar"", ""timestamp"": ""1683129843.042739"", ""content"": ""We just need to migrate liquibase change logs table separately to complete the cycle.""}, {""user"": ""askumar"", ""timestamp"": ""1683611590.475249"", ""content"": ""\nReceived an update on AWS ticket on DMS migration :\n\n\n```Thanks for your cooperation and patience while I worked on your case. Please allow me to extend an apology for the delay as I couldn't get back to you with my response earlier. I regret the inconvenience caused and will try to address all of your queries at the earliest.\n\nTo reiterate my understanding of your concern, you have an RDS for PostgreSQL instance with multiple schemas per services and you are planning to migrate the schema for a specific service from the RDS instance to an Aurora cluster. While migrating the data using DMS, only the data is migrated however, objects such as constraints, keys, index, sequences are not migrated. In this context, you have some queries and would like to get clarification on the same.\n\nIn order to address each of your query in detail, I've answered your questions below, one at a time. Please feel free to correct me at any point, if my understanding does not lie inline with your concerns.\n\n1. Migrating entire schema say A that should migrate the constraints, keys as well along with data.\n\nTo begin with, please allow me to inform you that, AWS DMS takes a minimalist approach and only creates target schema objects that are required to efficiently migrate data. As such, DMS only supports basic schema migration, including the creation of tables, primary keys, and in some cases unique indexes, but it doesn't automatically create any other objects that are not required to efficiently migrate the data from the source.\n\nTherefore, DMS does not automatically create on the target nor migrate objects such as secondary indexes, foreign keys, user accounts, non-primary key constraints, sequences, default values, stored procedures, triggers, synonyms, etc. that are not directly related to the table data.\n\nAlong with that, I would also like to add that it's usually recommended to drop primary key indexes, secondary indexes, referential integrity constraints, and data manipulation language (DML) triggers, or you can delay their creation until after the full load tasks are complete. You don't need indexes during a full load task, and indexes incur maintenance overhead if they are present. Because the full load task loads groups of tables at a time, referential integrity constraints are violated. Similarly, insert, update, and delete triggers can cause errors, for example if a row insert is triggered for a previously bulk loaded table. Other types of triggers also affect performance due to added processing.\n\nThis applies to all the engines that are supported by AWS DMS and is also documented in our AWS DMS official documentation [1].\n\nUnfortunately, this means that you will not be able to migrate such secondary objects using DMS, and these objects must be added manually on the target database, once the migration is completed. In your case, you can also consider using the database engine's native tools for such migrations such as logical dump using pg_dump and restore using pg_restore. However, I would like to highlight that even with engine native tools, existing indexes and foreign keys can cause significant delays during data restore. Dropping foreign keys, primary keys, and indexes before restore and adding them after restore can drastically reduce migration time.\n\nKindly refer the AWS Blog at [2] that discusses different strategies for migrating data from source PostgreSQL database to a target RDS/Aurora for PostgreSQL database\n\n2. Also can we snapshot migration where we only specify what schemas we want to be migrated?\n\nIn this query, I understand that you are talking about taking a snapshot of an RDS instance with only specified schemas and restore it to another instance. To answer this, please allow me to inform you that a snapshot of an RDS instance is a storage level backup of all the underlying volumes associated with the RDS instance and hence, a snapshot constitutes of data of an entire DB instance and not just individual databases or schemas.\n\nTherefore, to answer your question, it is not possible to create a snapshot of an RDS instance with only specific schemas. Similarly, it is not possible to migrate an RDS for PostgreSQL snapshot to an Aurora cluster with only specific schemas.\n\n3. Can multiple snapshots from different RDS instances be migrated to one target Aurora instance ?\n\nAs mentioned above, a snapshot is a volume level backup till a certain period in time and hence, while restoring the snapshot, the new instance that is restored from the snapshot would only have the data captured in the snapshot. Please note, you can't restore from a DB snapshot to an existing DB instance; a new DB instance is created when you restore.\n\nFor more details on RDS Snapshots and restore, kindly refer [3] and [4].\n\nSimilarly, during snapshot migration of an RDS PostgreSQL DB instance to an Aurora PostgreSQL DB cluster [5], the new Aurora PostgreSQL DB cluster is populated with the data from the original RDS for PostgreSQL DB instance.\n\nOnce an instance/cluster is restored or migrated from the snapshot, the data from the restored instance/cluster will be a copy of the original RDS PostgreSQL instance, till the point when snapshot was taken. Based on your use-case, using snapshot for migrating data from RDS PostgreSQL to Aurora may not be much helpful except from the fact that it will already have all the data in place that belongs to the desired schema. If the size of your database is not large and if feasible at your end, once the snapshot is migrated to an Aurora cluster, you can consider deleting data from all the other schemas and only retain the data and objects in the required schema. For this, following approach can be followed:\n\n1. Take snapshot of the source RDS for PostgreSQL instance.\n2. Migrate RDS for PostgreSQL instance snapshot to an Aurora cluster [5].\n3. Once the new Aurora cluster is created and in available state, connect to the database cluster.\n4. In the database cluster, drop all the other schemas and their data objects, that are not required.\n\nThe above approach is relatively simple and straightforward however, it may take longer based on the number of schemas that you have and if the size of data in those schemas is huge.\n\nWhile I might not have been of great assistance here due to the limitations and architecture of the service, I hope that the above information may have addressed your queries and that I was able to provide some clarity with your understanding. Please feel free to reach back, if you have any other queries/concerns. I will be glad to assist you better!```\n""}, {""user"": ""askumar"", ""timestamp"": ""1683611649.140839"", ""content"": ""Summary :\nAWS suggesting either adding constraints manually or using pg_dump approach.\n\nAlso multiple snapshots cannot be migrated to one instance""}, {""user"": ""aganivada"", ""timestamp"": ""1683612396.935049"", ""content"": ""thank you <@U03KLHDKL1H> so are we planning to go ahead with pg_dump? would pg_dump address both the use cases or do we have any challenges?""}, {""user"": ""askumar"", ""timestamp"": ""1683613719.550769"", ""content"": ""<@U02BV2DGUKC> pg_dump seems to be working fine for core.\nFor commons data base, it seems to be taking very long time in copying BLOBS, trying to find a way for it.""}, {""user"": ""aganivada"", ""timestamp"": ""1683614339.570349"", ""content"": ""Ok""}]" "1687851575.183979 ","[{""user"": ""aganivada"", ""timestamp"": ""1687851575.183979"", ""content"": ""<@U04JT69T00K> can you add <@U03KLHDKL1H> to stage and share creds with Ashwani? this is required for DB migration we also might need a task on your name to provision an aurora instance in stage. Once migration completes we can review IAM roles. cc: <@U026PMDB1ND>""}, {""user"": ""pjha"", ""timestamp"": ""1687851624.769859"", ""content"": ""sure <@U02BV2DGUKC>""}]" "1684570316.406549 ","[{""user"": ""aganivada"", ""timestamp"": ""1684570316.406549"", ""content"": ""<@U03KLHDKL1H> please check instances provisioned in int""}, {""user"": ""askumar"", ""timestamp"": ""1684570832.328479"", ""content"": ""We had stopped cluster not being used, deleting them now""}, {""user"": ""askumar"", ""timestamp"": ""1684573309.058939"", ""content"": ""There is one DB being used by <@U0336QZAF98> for testing.""}]" "1677240533.173869 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1677240533.173869"", ""content"": ""yes <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1678193857.524889"", ""content"": ""<@U0336QZAF98> can you share the FF name for this""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678195460.610219"", ""content"": ""<@U02BV2DGUKC> ""}, {""user"": ""aganivada"", ""timestamp"": ""1678195484.979809"", ""content"": ""thank you <@U0336QZAF98>""}]" "1688638530.550119 ","[{""user"": ""ppant"", ""timestamp"": ""1688638530.550119"", ""content"": "" Facing 401 unauthorised issue while setting up CI/CD pipeline for vms-askai-service \nI added the .ci-settings.xml file but still 401 is coming up\ncc: ""}, {""user"": ""ppant"", ""timestamp"": ""1688640284.855809"", ""content"": ""Had some issues with relative path and adding `.ci-settings.xml` fixed the 401 issue. Thanks <@U02BV2DGUKC>\nFor more context behind this - ""}]" "1682566271.384239 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1682566271.384239"", ""content"": ""<@U02BV2DGUKC> - Can you deploy estimation service 0.0.9.1 (hotfix) to stage cc: ""}, {""user"": ""svummidi"", ""timestamp"": ""1682567111.873989"", ""content"": ""<@U0336QZAF98> <@U02BV2DGUKC> 0.0.9.2 - There is a new version - If we are deploying can we use this version to include a fix for time line view?""}, {""user"": ""aganivada"", ""timestamp"": ""1682567263.800449"", ""content"": ""sure <@U040RCBPBEC> /<@U0336QZAF98> deploying 0.0.9.2 of estimation service to stage""}]" "1684218586.384089 ","[{""user"": ""anair"", ""timestamp"": ""1684218586.384089"", ""content"": ""<@U02BV2DGUKC> <@U0336QZAF98> we need to start prepping for another 1-2 okta integrations. how are we feeling about those""}, {""user"": ""aganivada"", ""timestamp"": ""1684220125.503909"", ""content"": ""<@U033PPLNFRU> I see the documentation work already done for okta, resharing articles added by <@U0336QZAF98>""}, {""user"": ""aganivada"", ""timestamp"": ""1684220135.212519"", ""content"": ""\n""}]" "1684685197.884949 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1684685197.884949"", ""content"": ""<@U02BV2DGUKC> <@U04JT69T00K> - just fyi.. made a couple of changes to latest-images script to get it to sort the output by date and also print more than one entry if needed.\n\n```latest-images.py --aws-profile ecrshared --excl-snapshots --count 2\n\n SERVICE RELEASE DATE RELEASE VERSION\n1: pulse-manager 2023-05-20 16:53:29 0.0.18\n2: 2023-05-20 16:52:17 0.0.18.1\n1: sentiment-collector 2023-05-19 13:54:34 0.0.18\n2: 2023-05-15 17:59:16 0.0.17.6\n1: dashboard-app 2023-05-19 13:19:04 0.0.16\n2: 2023-05-03 22:40:02 0.0.15.1```""}, {""user"": ""aganivada"", ""timestamp"": ""1684728463.459469"", ""content"": ""cool!! thank you <@U026PMDB1ND> this is very helpful as mostly it has been tribal knowledge and we had to go to gitlab a few times to verify builds""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684857160.430879"", ""content"": ""I updated admin-gw to name the builds with \""SNAPSHOT\"" for non-main branches. That way we identify release builds""}]" "1676974766.439079 ","[{""user"": ""ppant"", ""timestamp"": ""1676974766.439079"", ""content"": ""<@U02BV2DGUKC> Regarding different email templates for re-invitation for SSO migration, turns out Auth0 only supports one email template per use case (). Now, there are a couple of ways in which we can send an email based on a different template for re-invitation\n1. *Use a different template type for re-invitation -* There are some email templates that are not being used currently, like**Verification Email (using Code)**. We can use this and change the common variables in the template to match the invitation one. The con with this method is that from the backend we will have to call Auth0's verification API instead of invitation API in case of re-invitation. Also, not sure if we can generate the invitation link manually, couldn\u2019t find any API for this. Have asked this question \n2. *Update the same template and use Auth0\u2019s common variables to categorise email as invite or reinvite -* We can store the invite type in user metadata in Auth0 and read that in the email template. But we will need to update the email template such that the text correctly wraps around the invite type.\n""}, {""user"": ""aganivada"", ""timestamp"": ""1676994330.614059"", ""content"": ""sorry for late response <@U0431DZTPJM> just saw the message.\n\nI think (1) is very risky and we will be using **Verification Email (using Code)** for passwordless auth :slightly_smiling_face:\n\nfor (2) can we try app_metadata instead of user_metadata? both are associated with user object but advantage with app_metadata is it doesn't make it to tokens. how do we cleanup the data when user clicks on accept invitation?\n\ncan we also check in auth0 community of there is a better way to handle this? finally please check if there is anyway to disable the notification from auth0 side for this specific case (re-invite) and can we send emails from our notification service instead""}, {""user"": ""ppant"", ""timestamp"": ""1677061849.547309"", ""content"": ""We can use `user.app_metadata` variable and we will have to manage the data inside this when the user accepts the re-invite - such as re-invite accepted, failed etc. We can fire a cleanup request to auth0 on accepting the invite""}, {""user"": ""ppant"", ""timestamp"": ""1677062056.437729"", ""content"": ""We can disable the email notification from auth0 by marking `send_invitation_email=false` in the request body to invitation API but the caveat is there is no way to create the invitation link manually. So we can\u2019t insert the link in our email""}, {""user"": ""aganivada"", ""timestamp"": ""1677062423.923289"", ""content"": ""hmmm I think we can get invitation link by calling auth0 API""}]" "1683326838.929839 ","[{""user"": ""anair"", ""timestamp"": ""1683326838.929839"", ""content"": ""<@U0336QZAF98> EA is also interested in SSO - Okta, are we prepped there?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1683343943.310109"", ""content"": ""yes <@U033PPLNFRU>.. we did for okta ""}]" "1681917522.888679 ","[{""user"": ""askumar"", ""timestamp"": ""1681917522.888679"", ""content"": ""\nTesting data migration for core-stack in INT, responses might be delayed for api calls""}, {""user"": ""askumar"", ""timestamp"": ""1681921545.302549"", ""content"": ""Testing Done""}]" "1679671449.439519 ","[{""user"": ""ppant"", ""timestamp"": ""1679671449.439519"", ""content"": "" Created this how-to doc for some common issues with Poetry ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679679871.051399"", ""content"": ""Awesome. Thanks!""}]" "1692446281.934289 ","[{""user"": ""aganivada"", ""timestamp"": ""1692446281.934289"", ""content"": ""<@U026PMDB1ND> can we close the cdk release branch on monday? if we run into any issues we could use the release branch to make changes instead of adding a HF branch in cdk cc: <@U04JT69T00K>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692564095.003419"", ""content"": ""<@U02BV2DGUKC> - you mean you want to keep it open till Monday? If yes, I am okay with it""}, {""user"": ""aganivada"", ""timestamp"": ""1692591831.594089"", ""content"": ""yes <@U026PMDB1ND>, thank you.\n\n <@U04JT69T00K> lets close the cdk release branch today""}]" "1688998282.698019 ","[{""user"": ""sranjan"", ""timestamp"": ""1688998282.698019"", ""content"": "" <@U02BV2DGUKC> <@U03KLHDKL1H> Not able to access AWS .""}, {""user"": ""pjha"", ""timestamp"": ""1688998321.547109"", ""content"": ""Are you connected to VPN ?""}, {""user"": ""askumar"", ""timestamp"": ""1688998330.888049"", ""content"": ""Try with VPN""}, {""user"": ""sranjan"", ""timestamp"": ""1688998357.330429"", ""content"": ""Ok ..Thanks <@U04JT69T00K> <@U03KLHDKL1H>""}]" "1682495527.270939 ","[{""user"": ""bganganna"", ""timestamp"": ""1682495527.270939"", ""content"": ""<@U0336QZAF98> Can we document the step for reprocessing the open text comments in prod to populate the date so that it wont be missed cc <@U03DHUAJVMK>""}, {""user"": ""svummidi"", ""timestamp"": ""1682532012.192829"", ""content"": ""<@U03NZ7Z52S2> <@U0336QZAF98> I have one draft, I will updated it and publish today.""}, {""user"": ""svummidi"", ""timestamp"": ""1682532906.339669"", ""content"": """"}, {""user"": ""bganganna"", ""timestamp"": ""1682561432.319639"", ""content"": ""Thanks <@U040RCBPBEC>, Is there any release recipe doc we follow where we should update this step as post deployment requirement""}, {""user"": ""svummidi"", ""timestamp"": ""1682566495.243259"", ""content"": ""<@U03NZ7Z52S2> Usually the pattern is cover with feature flag. Update tenant specific comments, enable feature flag and then update all tenant comments and then enable flag for all tenants.\nWe don\u2019t want to do it part of the release recipe instead we want to do it as post release activity.""}]" "1680695120.610419 ","[{""user"": ""araman"", ""timestamp"": ""1680695120.610419"", ""content"": ""<@U026PMDB1ND> <@U033PPLNFRU> :\nBefore we make audit log on GA, couple of points:\n1. - Clean up old entries/records that are not categorised into sections in production. We were seeing records in stage with out any categorisation\n2. <@U026PMDB1ND> `Create And Verify Query AuditLog` failed in prod at step - *Query the audit log service for the log message and check that the names still remain resolved (via deletedobject index*)\ncc <@U02BV2DGUKC>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680709863.761019"", ""content"": ""Ok, will take a look at that test.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680746716.500009"", ""content"": ""This is due to deploying lambdas from develop :disappointed:""}, {""user"": ""aganivada"", ""timestamp"": ""1680754418.770599"", ""content"": ""<@U026PMDB1ND> me and <@U0431DZTPJM> were discussing this yesterday since we don't have versioning yet (Prabhu is working on it) would it make sense to release the lambda code during or post branch-cut? it will ensure that we have working version in main branch and also tagged.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680754448.471999"", ""content"": ""Yes. We absolutely should""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680754515.258089"", ""content"": ""We should also include branch names in the package names (just like we do with Java). That way develop would never overwrite what is in main etc.""}, {""user"": ""aganivada"", ""timestamp"": ""1680754691.093349"", ""content"": ""sure makes sense, we could also publish branch name as either env variable in lambda or in lambda name itself.\n\nfor now, we will go ahead and release all existing lambda functions manually before Prabhu merges his changes to develop branch""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680754828.093519"", ""content"": ""sure""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680755592.159429"", ""content"": ""<@U03DHUAJVMK> <@U033PPLNFRU> - this issue is now resolved.""}, {""user"": ""anair"", ""timestamp"": ""1680761788.400469"", ""content"": ""thanks <@U026PMDB1ND>, <@U03DHUAJVMK> are we ok with releasing this""}, {""user"": ""araman"", ""timestamp"": ""1680763010.687329"", ""content"": "" <@U033PPLNFRU> are we ok with this empty section records? <@U02BV2DGUKC> Can this be taken up?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680764093.300049"", ""content"": ""I think that should be okay""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680764148.081639"", ""content"": ""What do you say, <@U033PPLNFRU>. We just need to make sure that we'd never see system user.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680766041.313299"", ""content"": ""<@U03DHUAJVMK> can you share the creds where you see these records?""}, {""user"": ""anair"", ""timestamp"": ""1680798667.113329"", ""content"": ""<@U026PMDB1ND> i think we definitely cannot see system_user""}, {""user"": ""anair"", ""timestamp"": ""1681275338.775409"", ""content"": ""hi <@U026PMDB1ND> has the system_user issue been fixed cc <@U03DHUAJVMK>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681313843.703469"", ""content"": ""Was there a system user issue, @anu?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681313917.411319"", ""content"": ""<@U033PPLNFRU> not sure which issue are you referring to.""}]" "1690868762.739819 ","[{""user"": ""pjha"", ""timestamp"": ""1690868762.739819"", ""content"": ""<@U02BV2DGUKC> <@U03KLHDKL1H> RDS Proxy reader endpoint is connecting to both the underlying reader replica of the RDS cluster.""}, {""user"": ""pjha"", ""timestamp"": ""1690868821.816469"", ""content"": ""checking the activity on the both the reader endpoint.""}, {""user"": ""pjha"", ""timestamp"": ""1690868951.417259"", ""content"": ""Here I am connecting querying to the reader endpoint of PROXY""}, {""user"": ""pjha"", ""timestamp"": ""1690869086.460729"", ""content"": ""whereas no activity for the query on the RDS writer endpoint""}, {""user"": ""aganivada"", ""timestamp"": ""1690872378.848199"", ""content"": ""nice <@U04JT69T00K>, is it round-robin WRT routing requests? do we see that the requests are equally distributed?""}, {""user"": ""pjha"", ""timestamp"": ""1690873012.736959"", ""content"": ""metrics pattern looks same for both the reader instance""}, {""user"": ""aganivada"", ""timestamp"": ""1690873175.436859"", ""content"": ""ok cool, <@U03KLHDKL1H> based on yesterdays discussion can we force reader instance from property/env-var if proxy is enabled?""}, {""user"": ""askumar"", ""timestamp"": ""1690873391.360199"", ""content"": ""Yes <@U02BV2DGUKC> the environment vars can be accessed and based on the flag we can decide whether to populate proxy or the actual details using cluster name.\nThis would need work in the library to change the logic.""}, {""user"": ""aganivada"", ""timestamp"": ""1690873441.039079"", ""content"": ""ok lets see if we have sufficient bandwidth we can pick it up in next sprint?""}, {""user"": ""askumar"", ""timestamp"": ""1690873478.030299"", ""content"": ""sure <@U02BV2DGUKC>... will add task for it""}, {""user"": ""aganivada"", ""timestamp"": ""1690873484.552829"", ""content"": ""meanwhile can we route other services (where read-only db is not set yet) to proxy?""}, {""user"": ""askumar"", ""timestamp"": ""1690873567.530099"", ""content"": ""yeah that can be done, only Prashant's CDK changes would be sufficient for that.""}, {""user"": ""aganivada"", ""timestamp"": ""1690873681.557229"", ""content"": ""<@U04JT69T00K> which services are currently most active in int?""}, {""user"": ""pjha"", ""timestamp"": ""1690873882.430319"", ""content"": ""currently pulse manager seems to be most active in int""}, {""user"": ""aganivada"", ""timestamp"": ""1690873999.238809"", ""content"": ""ok lets discuss on how to add proxy to services tmrw during planning. meanwhile can you submit MR for your changes and turn off proxy <@U04JT69T00K>? lets take a call on merging MR based on any chances fro regression""}, {""user"": ""pjha"", ""timestamp"": ""1690874025.913779"", ""content"": ""sure <@U02BV2DGUKC>""}, {""user"": ""askumar"", ""timestamp"": ""1690874028.030089"", ""content"": ""cool""}, {""user"": ""aganivada"", ""timestamp"": ""1690874090.446699"", ""content"": ""<@U04JT69T00K> regarding the change we can decide to enable proxy by service and env correct?""}, {""user"": ""pjha"", ""timestamp"": ""1690874471.486569"", ""content"": ""<@U02BV2DGUKC> yes we decide by service and env""}, {""user"": ""pjha"", ""timestamp"": ""1690874488.700779"", ""content"": ""basically I have added in the cdk.json whether to connect to proxy or not""}]" "1683263777.157669 ","[{""user"": ""ppant"", ""timestamp"": ""1683263777.157669"", ""content"": "" For backend support for multiple instance deployment, we have made some changes in the openApi generated ApiClient which will support reading endpoint configurations. Currently for PoC, we are doing it via application.yml but can move it to SSM so that we won\u2019t have to update the yml file of each service. Configs for this look like this -\n\n```multi-instance-config:\n basePathMappings:\n - tenantId: 12345\n services:\n - name: core-data-service\n basePath: \n - name: onboarding-service\n basePath: \n\n - tenantId: 54321\n services:\n - name: core-data-service\n basePath: \n - name: tenancy-service\n basePath: ```\nBased on tenant IDs, we will be routing the request. If the a service name or tenant ID is not present here, the request will go to the default endpoint of the environment.\n\nHere is the MR containing these changes - ""}, {""user"": ""aganivada"", ""timestamp"": ""1683263833.064229"", ""content"": ""thank you <@U0431DZTPJM>, can we also validate the flow with reactive app as well?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683265569.018089"", ""content"": ""<@U0431DZTPJM> The port value cannot go beyond 65535, right? Why are we tying the port (which is of short type) to tenantId (which is of long type)?""}, {""user"": ""ppant"", ""timestamp"": ""1683265611.297369"", ""content"": ""<@U026PMDB1ND> This is just for PoC. I was just trying to see if the endpoint changes according to the tenant IDs so I put some random values there""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683265800.935619"", ""content"": ""Ok, cool/""}, {""user"": ""ppant"", ""timestamp"": ""1683270165.967719"", ""content"": ""<@U02BV2DGUKC> Tried this with onboarding app by explicitly setting the tenant IDs in thread context. Seems to work fine""}, {""user"": ""aganivada"", ""timestamp"": ""1683271501.872059"", ""content"": ""cool thank you <@U0431DZTPJM>""}]" "1680493987.220499 ","[{""user"": ""aganivada"", ""timestamp"": ""1680493987.220499"", ""content"": ""<@U0336QZAF98> starting a new thread to discuss on the Haleon SSO setup. Looking at decoded SAML-response they seem to be using SignatureMethod = rsa-sha256 and DigestMethod = sha256 should we configure the same in SSO or does auth0 pick these by default?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1680494160.587079"", ""content"": ""I think this picks up by default.i think this is something related to certificate""}, {""user"": ""aganivada"", ""timestamp"": ""1680494517.017009"", ""content"": ""ok. I saw the same x509 cert they shared with us in SAML response so was wondering if something is getting wrong with the algorithm chosen for calculating digest""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680572925.980339"", ""content"": ""Does auth0 not log anything?""}, {""user"": ""aganivada"", ""timestamp"": ""1680581550.367229"", ""content"": ""yes <@U026PMDB1ND> auth0 error log stated that certificate digest dont match but the same flow worked fine when customer attempted last night\n\n> Anil,\n> \n> I think this is a configuration issue \u2013 we need to validate once with them what they have done at their end\n> \""description\"": \""Invalid thumbprint (configured: 890F91596398F355B4E927F9762317A2CA1FC9B6. calculated: D8C105AEA200750F9296105B7E602AA0FA3E6126)\"",\n> \n> \n> \n> \n> Thanks\n> Seshan""}]" "1683090393.068739 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1683090393.068739"", ""content"": "" - Came to know that today is <@U03DHUAJVMK>'s 1-year anniversary at Axiamatic :confetti_ball: :tada:\n\nCongratulations, Anu!""}, {""user"": ""aganivada"", ""timestamp"": ""1683107719.435209"", ""content"": ""Congratulations <@U03DHUAJVMK>!! :clap::clap:""}]" "1681108461.234639 ","[{""user"": ""ppant"", ""timestamp"": ""1681108461.234639"", ""content"": "" Created this doc back in January for refactoring Tenancy , will be using it for keeping track of all the refactoring related tasks and notes. <@U02BV2DGUKC> Also I am thinking of following core-data-esque delegate-helper design pattern while refactoring as there are a couple of delegate methods that are called directly from other services""}, {""user"": ""aganivada"", ""timestamp"": ""1681110409.182019"", ""content"": ""ack <@U0431DZTPJM>, thank you for writing this up. I can take up usersapiservicedelegate and VendorApiServiceDelegate for refactoring""}]" "1679002147.216069 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1679002147.216069"", ""content"": ""<@U02BV2DGUKC> <@U04JT69T00K> - I don't think we should look at PLAT-1589 (automatic deployment of build to int) until we are done with \""add ability to deploy multiple instances of a service during development\"", unless we want to only target \""develop\"" branch builds.\n\nIt would make sense to auto-deploy builds if have support for multi instance deployment as it wouldn't break someone else's test.\n\nAlso, did we figure out how to get apps to talk to multiple backend instances (of say tenancy) from the same apps instance by using a header (potentially set on the frontend)? To me it is a critical requirement in this story.""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1679023143.514949"", ""content"": ""<@U026PMDB1ND>, <@U04ARFN3WQP> is looking in to adding header from ui code based on local storage item. will update once we test it out.""}, {""user"": ""aganivada"", ""timestamp"": ""1679023821.692149"", ""content"": ""> did we figure out how to get apps to talk to multiple backend instances (of say tenancy) from the same apps instance by using a header (potentially set on the frontend)? To me it is a critical requirement in this story.\n<@U026PMDB1ND> from apps and other backend services side this seems like a mjor issue since backend domain is mapped too deep sometimes in library and sometimes in application.yml. so instead of researching this at spring level we are planning to research if we can do something on ALB side to route requests based on header \n\nso if apps can forward the header and if we can use ALB to route request to specific instance based on the header it might help routing without major changes. cc: <@U04JT69T00K>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679024152.601919"", ""content"": ""Sounds like an interesting approach""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679024230.883139"", ""content"": ""Since we deploy ALB with service stacks wonder if this is doable. Would be interesting to see if this pans out.""}, {""user"": ""aganivada"", ""timestamp"": ""1679024536.892079"", ""content"": ""yeah we need to try this out manually to check if it is possible and then try CDK route.""}]" "1694601295.005059 ","[{""user"": ""aganivada"", ""timestamp"": ""1694601295.005059"", ""content"": ""<@U05D3HNDW3D> I am still getting error on using /axmd-oc command, verified that both AXMD_SLACK_BOT_TOKEN, AXMD_SLACK_ONCALL_CHANNEL_ID keys are added as env variables to admin-gw in int. cc: <@U026PMDB1ND>""}, {""user"": ""bkathi"", ""timestamp"": ""1694614959.454429"", ""content"": ""that\u2019s weird it was working on monday""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1694615082.699149"", ""content"": ""Can we directly invoke the end point via curl to see where it is failing?""}, {""user"": ""bkathi"", ""timestamp"": ""1694615153.498609"", ""content"": ""looks like the whitelist is not working getting `{\""message\"":\""Unauthorized\""}` response""}, {""user"": ""bkathi"", ""timestamp"": ""1694615170.261419"", ""content"": ""here is the curl command\n```curl -d \""trigger_id=T0001\"" -X POST ```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1694615234.597169"", ""content"": ""Oh.. Anil can you deploy from the release branch again? I must have deployed from the develop branch as part of some validation ""}, {""user"": ""aganivada"", ""timestamp"": ""1694616797.231319"", ""content"": ""sure <@U026PMDB1ND>""}, {""user"": ""aganivada"", ""timestamp"": ""1694619404.288059"", ""content"": ""<@U05D3HNDW3D> slack commands are working now, there is also a message with \""null\"" is this expected? cc: <@U026PMDB1ND>""}, {""user"": ""bkathi"", ""timestamp"": ""1694625292.792219"", ""content"": ""Yes, it\u2019s not letting me set an empty response (response from the endpoint is directly printed out there), so maybe i will replace it with \u201cLaunching dialog\u2026\u201d or something like that""}]" "1684394132.359469 ","[{""user"": ""aganivada"", ""timestamp"": ""1684394132.359469"", ""content"": ""<@U04JT69T00K> we are noticing one strange issue with GL-runner only for prod auth redirects are not working when we run system_tests but for other env's it works fine. Let me know when you have few mins to check this cc: <@U03NZ7Z52S2>""}, {""user"": ""bganganna"", ""timestamp"": ""1684394173.389229"", ""content"": ""cc <@U03DHUAJVMK>""}, {""user"": ""pjha"", ""timestamp"": ""1684394271.411819"", ""content"": ""<@U02BV2DGUKC> can we check now""}, {""user"": ""aganivada"", ""timestamp"": ""1684394381.700389"", ""content"": ""sure""}, {""user"": ""aganivada"", ""timestamp"": ""1684394511.229159"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1684394850.814129"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1684395944.027009"", ""content"": ""<@U03NZ7Z52S2> changing to GL-runner for testing. will revert once done""}, {""user"": ""bganganna"", ""timestamp"": ""1684395980.079109"", ""content"": ""sure <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1684397827.965299"", ""content"": ""<@U03NZ7Z52S2> this seems to be happening because we enabled bot-protection in prod and for some reason auth0 is thinking that the calls from GL-runner are coming from a bot. we are checking if we can whitelist ip on auth0""}, {""user"": ""aganivada"", ""timestamp"": ""1684398609.879269"", ""content"": ""seems to be working now, will trigger a sanity on prod one this is done \n\nTurns out auth0 was prompting captcha to headless browser and headlessbrowser was like ""}, {""user"": ""bganganna"", ""timestamp"": ""1684398657.301169"", ""content"": ""sure""}]" "1683591108.496209 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1683591108.496209"", ""content"": ""<@U0431DZTPJM> - I am trying to do something along these lines from a service that depends on platform_libs_epoch:\n\n`url = self.app_config.axm_service_urls().get(\""core-data\"", None)`\nor\n```url = CoreDataService(self.app_config, self.token_api).core_data_url```\nAnd I see:\nFileNotFoundError: [Errno 2] No such file or directory: 'platform_libs_epoch/configuration/axm_services.json'\n\nIs the file expected to be the package?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683592676.600329"", ""content"": ""Also, platform_libs_epoch/common/axm_http probably needs some tweaks. Wonder if we never ran into a failing case?\n\n``` response = axm_http.get(\n File \""/Users/rama/axm/apps-services-admin-gw/.venv/lib/python3.9/site-packages/platform_libs_epoch/common/axm_http.py\"", line 15, in get\n raise AxmAPIException(\nplatform_libs_epoch.common.axm_api_exception.AxmAPIException: Failed to perform api call\n--- Logging error in Loguru Handler #1 ---\nRecord was: None\nTraceback (most recent call last):\n File \""/Users/rama/axm/apps-services-admin-gw/.venv/lib/python3.9/site-packages/loguru/_handler.py\"", line 270, in _queued_writer\n message = queue.get()\n File \""/Users/rama/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/queues.py\"", line 367, in get\n return _ForkingPickler.loads(res)\n File \""/Users/rama/axm/apps-services-admin-gw/.venv/lib/python3.9/site-packages/loguru/_recattrs.py\"", line 77, in _from_pickled_value\n value = pickle.loads(pickled_value)\nTypeError: __init__() missing 5 required positional arguments: 'url', 'http_method', 'status_code', 'raw_request', and 'raw_response'\n--- End of logging error ---```""}, {""user"": ""ppant"", ""timestamp"": ""1683606226.551069"", ""content"": ""<@U026PMDB1ND> Yes the axm_services.json file is in the configuration directory of epoch ""}, {""user"": ""ppant"", ""timestamp"": ""1683606241.653789"", ""content"": ""Are you calling this from somewhere outside src? Like tests?""}, {""user"": ""ppant"", ""timestamp"": ""1683606578.122059"", ""content"": ""For the axm_http, I was calling it like this\n```response = axm_http.get(\n session=self.app_config.get_http_session(),\n url=url,\n headers=axm_http.get_headers_with_token(self.token_api.get_token()),\n data=f\""tenant_id-{tenant_id}\"",\n component=f\""{self.__class__.__name__}.get_pvt_api\"",\n )```\nYou need to pass the session, headers and other info to this method. Basically it is just a wrapper over requests.get to allow better logging and exception handling""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683640311.896619"", ""content"": ""<@U0431DZTPJM> - I was invoking it from a different service""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683640380.500579"", ""content"": ""How does this work when invoked from the service invoker lambda?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683640504.906069"", ""content"": ""The issue with axm_http is how it works if the response is not 200 Ok""}, {""user"": ""ppant"", ""timestamp"": ""1683640944.009159"", ""content"": ""<@U026PMDB1ND> Currently for service invoker lambda, we are calling the tenant_deletion method of each service like this. Did this to create a more common approach as all the other APIs in other lambdas were called this way\n\n```def call_axm_service(event_type, service_name, tenant_id, mode, event_id):\n if not event_type == TENANT_DELETION:\n raise ValueError(f\""Unexpected event type {event_type}\"")\n\n service = axm_services[service_name](app_config, token_api)\n try:\n return service.tenant_deletion_coordination(tenant_id=tenant_id, mode=mode, event_id=event_id)\n ```\nUsing the `axm_service_urls`, this was how it was done ""}, {""user"": ""ppant"", ""timestamp"": ""1683640988.163129"", ""content"": ""Hmm, let me look at the axm_http issue. Generally if the response is not ok, we are raising AxmException\n``` if not response.ok:\n raise AxmAPIException(\n component=component,\n url=url,\n http_method=\""GET\"",\n status_code=response.status_code,\n raw_request=data,\n raw_response=response.text,\n )```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683641250.365549"", ""content"": ""Hmm ok.. so if we move away from the json approach and move the constants into a python dict wouldn\u2019t it help make the calls from any client of that library?""}, {""user"": ""ppant"", ""timestamp"": ""1683641429.610559"", ""content"": ""Ideally the json should have been incorporated into the package because that was how initially coord lambda was deployed, using axm_service_urls to form endpoints. But yes, we can make the json into a python dict as it contains only the names of axm services.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683641516.108559"", ""content"": ""Or make loading of json work so that we can invoke this code from any client of the library ""}, {""user"": ""ppant"", ""timestamp"": ""1683641762.461099"", ""content"": ""Sure. Can you point me to the project repo\u2019s from where this is being called? I can look at it""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683645579.475849"", ""content"": ""Can you make it work from any lambda?""}, {""user"": ""ppant"", ""timestamp"": ""1683646036.468759"", ""content"": ""Sure, doing it for coordination lambda""}, {""user"": ""ppant"", ""timestamp"": ""1683648242.565019"", ""content"": ""<@U026PMDB1ND> Deployed coordination invoker lambda with these changes (here is the temporary MR for this ). Seems to be working fine. Here are the logs of the step function which succeeded ""}]" "1692163734.970809 ","[{""user"": ""aganivada"", ""timestamp"": ""1692163734.970809"", ""content"": ""<@U0431DZTPJM> do you see any issue with adding a default with error response in this method?""}, {""user"": ""ppant"", ""timestamp"": ""1692163870.913039"", ""content"": ""We can add it here, but if I remember, the default invitation type as part of the API specs is always INVITE (the one in which Auth0 sends email). So the default block won\u2019t be executed. But we can add it just to be on the safe side""}, {""user"": ""aganivada"", ""timestamp"": ""1692165857.888629"", ""content"": ""got it, so we are controlling this at API level?""}, {""user"": ""ppant"", ""timestamp"": ""1692166051.566859"", ""content"": ""Yup, the default will always be INVITE here""}]" "1676064650.127899 ","[{""user"": ""akalyta"", ""timestamp"": ""1676064650.127899"", ""content"": ""TL;DR: stuck a bit, need suggestion on how to add a secret key to CDK and make sure it is passed to application on int\n\nWondering if someone can help me: I am trying to make sure that OpenAI secret API key is passed from to the application on int. We currently have it stored in secrets manager. Anil suggested me yesterday to check `base-infra/bootstrap/setup-basics.sh` script and add the necessary functions here mimicking the existing ones for other services but I found the functions for OpenAI secret already present and unfortunately it seems that the secret key is still not being retrieved and passed to the application.\n\nI\u2019ve added the \u201cOPENAI_KEY\u201d string to `env-vars` in `estimation-service/cdk.json` __global __ (mimicking `core-data-service`), maybe it should fix the issues? I didn\u2019t push changes yet because I am not sure about pushing rules for `base-infra`. (second screenshot below)\n\nI also found the `base-infra/bootstrap/setup-functions.sh` script with this function (first screenshot below) which seems to be recording secrets but I presume this function is being run each time there\u2019s a deployment? Should I run it manually? Or am I looking in the wrong place?\n\nWould really appreciate a suggestion :)\ncc <@U026PMDB1ND> <@U040RCBPBEC> <@U02BV2DGUKC>""}, {""user"": ""svummidi"", ""timestamp"": ""1676071320.666909"", ""content"": ""<@U02Q02G09QX> setup-functions.sh is to add secret to the secrets manager, it is already executed in INT. You just need to make changes in estimation service scripts.""}, {""user"": ""akalyta"", ""timestamp"": ""1676071375.899839"", ""content"": ""thank you! To test if adding the \u201cOPENAI_KEY\u201d worked, can I make an MR for base infra?""}, {""user"": ""svummidi"", ""timestamp"": ""1676071457.238929"", ""content"": ""<@U02Q02G09QX> - I did not do anything now, I did this two weeks back. Have you tried executing your changes to the service and working. If working submit an MR.""}, {""user"": ""akalyta"", ""timestamp"": ""1676071978.903549"", ""content"": ""<@U040RCBPBEC> when i try executing the setup-functions script and provide OpenAI key, it said that it already exists - I am assuming that you have provided it. Now I added OPENAI_KEY into estimation cdk.json file and I think i need to push the change to see if it works, right?""}, {""user"": ""svummidi"", ""timestamp"": ""1676072061.398399"", ""content"": ""can you huddle?""}, {""user"": ""akalyta"", ""timestamp"": ""1676072075.307219"", ""content"": ""yes :)""}, {""user"": ""aganivada"", ""timestamp"": ""1676276631.255789"", ""content"": ""<@U02Q02G09QX> is this resolved now?""}, {""user"": ""akalyta"", ""timestamp"": ""1676300729.246579"", ""content"": ""Yes, thank you <@U02BV2DGUKC>!""}]" "1689983654.559019 ","[{""user"": ""svummidi"", ""timestamp"": ""1689983654.559019"", ""content"": ""<@U0336QZAF98> <@U0431DZTPJM> - Today <@U02Q02G09QX> faced an issue with deserializing one of the core data response for UserAttributesDTO\nIn the spec it is defined as below\n``` \""UserAttributesDTO\"": {\n \""type\"": \""object\"",\n \""properties\"": {\n \""metadata\"": {\n \""type\"": \""object\"",\n \""additionalProperties\"": {\n \""type\"": \""object\""\n }\n }\n }\n }```\nIn the generated code, translated as below but it is failing if the value for metadata is a string\n```metadata: Optional[Dict[str, Dict[str, Any]]] = None```\nFor example if the metadata is `{'tower': 'Digital & Commercial', 'location': 'APAC', 'department': 'Department Y'}` - It is not able to parse\n\nI think it is expected to generate the metadata type as below. Let me know if any of you have more details about this problem.\n```metadata: Optional[Dict[str, Any]] = None```""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690058030.788189"", ""content"": ""<@U040RCBPBEC> - in java - below is generated one - I think in Python - bcoz object is always typed to Dict<str,Any> - i think this issue is happening - If we change to - Optional[Dict[str, Any]] did it work ?\n\n```@JsonTypeName(\""UserAttributes\"")\n@Generated(value = \""org.openapitools.codegen.languages.SpringCodegen\"", date = \""2023-07-21T20:57:37.750840+05:30[Asia/Kolkata]\"")\npublic class UserAttributesDTO {\n\n @Valid\n private Map<String, Object> metadata = new HashMap<>();```""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690060270.598279"", ""content"": ""Instead of declaring additionalproperties as type object , if we have defined in the following way , it would have generated metadata: Optional[Dict[str, Any]] - We have two options - 1) modify in core openapi json 2) Or modify the copy of json which are having as stopgap solution\n``` \n \""UserAttributesDTO\"": {\n \""type\"": \""object\"",\n \""properties\"": {\n \""metadata\"": {\n \""type\"": \""object\"",\n \""additionalProperties\"": true\n }\n }\n }```""}, {""user"": ""svummidi"", ""timestamp"": ""1690220103.917559"", ""content"": ""<@U0336QZAF98> If the purpose of the flag allows non dictionary type objects like Strings as values, I think we need to change the core-data spec.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690220813.446309"", ""content"": ""<@U040RCBPBEC> will check with <@U028EDANJM9> on this and get back to you""}]" "1686724304.538569 ","[{""user"": ""pkarthikeyan"", ""timestamp"": ""1686724304.538569"", ""content"": "" Once a while we see the JDBC connection error in token-manager (). We see even the Spring health endpoint warning that the health check took time beyond the threshold (). A similar observation is also seen in lightstep. (, ). We see a similar behavior in score-provider too. We would need some help to identify the root cause of the issue. cc: <@U02D4DUKDQC>""}, {""user"": ""ppant"", ""timestamp"": ""1686742094.400959"", ""content"": ""From what we were able to access, the connection pool was maxed out for them as the query took longer time to execute and kept hold of threads. One thing we can try to do is increase the max limit of connection pool size. Also, other than the query in `findClientRegistrationByPrincipalName`, are there any other queries which are taking longer to execute or erroring out?""}, {""user"": ""pkarthikeyan"", ""timestamp"": ""1686761483.514289"", ""content"": ""Can you share the details on how we say connection pool was maxed out. From lightstep I could see idle connections at 10""}, {""user"": ""pkarthikeyan"", ""timestamp"": ""1686761837.316429"", ""content"": ""The below is the query that gets executed. and it uses index scan.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686765392.925189"", ""content"": ""<@U03BPNY5AGM> - do we see this in prod & INT as well?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686765887.335829"", ""content"": ""We should correlate this with incoming requests into TM.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686780043.478739"", ""content"": ""Regex based lookup is not working properly in stage. Have been working with Logz all day.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686780532.067049"", ""content"": ""Though the number of TM requests were higher during the time of the error than the nearby times, TM is able to field a lot more requests at other times without any errors.""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1686797682.267049"", ""content"": ""<@U026PMDB1ND> we tried to correlate with hikari connection pool metrics and there was no spike in used connections at that point so it was not clear why a new connection was being setup""}, {""user"": ""pkarthikeyan"", ""timestamp"": ""1686798673.013199"", ""content"": ""<@U026PMDB1ND> I did see an occurrence in PROD as well last week. But due to the log retention limit of 1 month we lost that.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686800668.105469"", ""content"": ""It is possible that RDS had a hiccup""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1686802874.930579"", ""content"": ""we can enable debug logging for hikari and see if it is not too noisy and get any further info but I suspect this is a transient network issue too. Not sure if there are any metrics for fargate to track network failures""}]" "1686879891.330729 ","[{""user"": ""aganivada"", ""timestamp"": ""1686879891.330729"", ""content"": ""<@U04JT69T00K> is there a way to switch pipeline to shared runner automatically if the job is stuck for say more than x minutes?""}, {""user"": ""pjha"", ""timestamp"": ""1686891838.239289"", ""content"": ""not sure <@U02BV2DGUKC> ,I don't think there would be a way to switch in the running pipeline, I will still look into this .""}, {""user"": ""aganivada"", ""timestamp"": ""1686891947.624789"", ""content"": ""Yeah I agree let's setup sometime and discuss next week on this. BTW Do we have documentation on reverting the changes so builds can run back on shared runner?""}, {""user"": ""pjha"", ""timestamp"": ""1686892002.148789"", ""content"": ""yes :""}, {""user"": ""pjha"", ""timestamp"": ""1686892015.841289"", ""content"": """"}]" "1689221728.497939 ","[{""user"": ""pjha"", ""timestamp"": ""1689221728.497939"", ""content"": "" Please review postgres engine version upgrade ""}, {""user"": ""pjha"", ""timestamp"": ""1689222002.111079"", ""content"": ""here, Strategy1 is as per the AWS recommendation of *Performing an upgrade dry run.*""}, {""user"": ""aganivada"", ""timestamp"": ""1689226438.082399"", ""content"": ""thank you <@U04JT69T00K>, for strategy 1 until we reach step 9 I guess we wont have any downtime?\n\nfor some of the checks listed in the page can we test 3 prod db clusters to check for any inconsistencies?""}, {""user"": ""pjha"", ""timestamp"": ""1689227005.587249"", ""content"": ""we don't need any downtime until we reach to step 9 and also can perform test on the prod DB for any inconsistencies.""}]" "1681315106.697039 ","[{""user"": ""aganivada"", ""timestamp"": ""1681315106.697039"", ""content"": """"}, {""user"": ""ppant"", ""timestamp"": ""1681315260.747139"", ""content"": ""I faced a similar issue couple of months back with tenancy\u2019s tenant rollback due to duplicate name ""}, {""user"": ""aganivada"", ""timestamp"": ""1681318865.652769"", ""content"": ""yeah makes sense <@U0431DZTPJM> , would it work if we split to helper class and used annotation on the helper?""}, {""user"": ""ppant"", ""timestamp"": ""1681319099.540769"", ""content"": ""Yeah that should fix this. Tried this earlier but ran into a major refactoring sort of rabbit hole where a delegate service was being called straightaway so dropped it at that time ""}, {""user"": ""aganivada"", ""timestamp"": ""1681319235.128419"", ""content"": ""ok cc: <@U03KLHDKL1H>""}]" "1678777462.829639 ","[{""user"": ""ppant"", ""timestamp"": ""1678777462.829639"", ""content"": "" Here is the doc detailing the options for lambda integrations with backend services in VPC - ""}, {""user"": ""aganivada"", ""timestamp"": ""1678783761.226149"", ""content"": ""thank you <@U0431DZTPJM>, considering the changes and extra cost involved in routing requests through private api-gw I think option 2 is better""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678785058.348469"", ""content"": ""<@U0431DZTPJM> <@U02BV2DGUKC> - quick question - interaction between microservices - does it happen via api gateway or via vpc private endpoint ?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678785076.874749"", ""content"": ""e,g when tenancy is calling core-data""}, {""user"": ""aganivada"", ""timestamp"": ""1678785098.051699"", ""content"": ""between microservice is through private domain we dont go via api-gw""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678785160.494099"", ""content"": ""i am assuming when you say private domain - it follows - <service-name>.services.<env>.exp-pinstripe.local - this format ?""}, {""user"": ""aganivada"", ""timestamp"": ""1678785173.271459"", ""content"": ""yup that is correct""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678785186.844349"", ""content"": ""got it..option 2 make sense""}]" "1690365115.404429 ","[{""user"": ""aganivada"", ""timestamp"": ""1690365115.404429"", ""content"": ""<@U0431DZTPJM> do we have multi-instance by tenant id enabled in all apps in int? I added the SSM parameter for redirecting tenancy calls to a custom instance but the calls still seem to be going to the actual instance""}, {""user"": ""ppant"", ""timestamp"": ""1690365729.032219"", ""content"": ""Not yet anil, haven\u2019t committed the changes in app services. Only tenancy and core have right now""}, {""user"": ""ppant"", ""timestamp"": ""1690365757.001829"", ""content"": ""I think I might have changes stashed for all Axiamatic services. Will take me max 2 hours to make for all services""}, {""user"": ""aganivada"", ""timestamp"": ""1690365829.437299"", ""content"": ""ok thank you <@U0431DZTPJM>, lets hold it may be until we have the branch cut done just to avoid any regression at this point.""}, {""user"": ""ppant"", ""timestamp"": ""1690365845.973949"", ""content"": ""Ok sure""}]" "1680615325.207649 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1680615325.207649"", ""content"": ""<@U040RCBPBEC> <@U03NZ7Z52S2> <@U03DHUAJVMK> - i have enabled soft delete feature flag for below tenants - 8677 12944 14037 - Thanks to <@U02BV2DGUKC> - i was able to veriy EA loading fine and anu test tenant . Let me know if you see any issues""}, {""user"": ""aganivada"", ""timestamp"": ""1680618537.412689"", ""content"": ""<@U0336QZAF98> please review if this is anyway related to the FF ""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1680618597.284839"", ""content"": ""Actually 404 error on pulse manager - it is not related to feature flag on scoring""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1680618674.029789"", ""content"": ""The error was no pulses found for a specific product instance""}, {""user"": ""bganganna"", ""timestamp"": ""1680618712.868339"", ""content"": ""<@U0336QZAF98> u enabled the flag or excluded some instances as well?""}, {""user"": ""bganganna"", ""timestamp"": ""1680618755.459499"", ""content"": ""> The error was no pulses found for a specific product instance\nyes seeing some err from PM side , checking with <@U02SF36PVKL> and <@U03RSS0S76Y>""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1680618874.588889"", ""content"": ""For 8677 - I saw following pulse instances excluded - 6394,8669,8668 <@U03NZ7Z52S2>""}]" "1687846600.854389 ","[{""user"": ""pjha"", ""timestamp"": ""1687846600.854389"", ""content"": "" We have TTL for the stack, if the stack is deleted but the SSM mapping still exist this can create an issue from the backend.\nEg.\n*/int/multi-instance-config/45814 : {\""multiInstanceConfigs\"": [{'serviceName': 'core_data-service', 'version': 'plat-2', '__user': 'prashantjha', '__date': '2023-06-27 10:31:22'}, {'serviceName': 'tenant-service', 'version': 'plat-2', '__user': 'prashantjha', '__date': '2023-06-27 11:06:43'}]}*\n\nNow lets say *tenant-service-plat-2* stack is deleted but the entry is not removed from the parameter this will cause issue since backend service will look for *tenant-service-plat-2* *endpoint.*\nShould we integrate it with the stack deletion lambda-function ?""}, {""user"": ""ppant"", ""timestamp"": ""1687846907.700589"", ""content"": ""IMO, makes sense to delete the SSM config for tenant ID. We can always get the previous value of that tenant\u2019s SSM parameter from slack messages and restore it, if required.""}, {""user"": ""pjha"", ""timestamp"": ""1687847214.533479"", ""content"": ""<@U0431DZTPJM> we will not be deleting the SSM Parameter, just need to remove the service-information from the parameter\nin above example we will just be removing *{'serviceName': 'tenant-service', 'version': 'plat-2', '__user': 'prashantjha', '__date': '2023-06-27 11:06:43'}* from the */int/multi-instance-config/45814* parameter .""}, {""user"": ""aganivada"", ""timestamp"": ""1687847794.376559"", ""content"": ""<@U04JT69T00K> /<@U0431DZTPJM> are we planning to add TTL date in SSM parameter also through CDK? if yes the can we update library to skip if the ttl crossed?""}, {""user"": ""ppant"", ""timestamp"": ""1687848002.187589"", ""content"": ""TTL date is in CDK.json. We can\u2019t put it in SSM because then from the backend we will have to check which service has died and skip its call""}, {""user"": ""aganivada"", ""timestamp"": ""1687848243.343369"", ""content"": ""hmmm so we have to cleanup/update via lambda during cleanup I guess""}]" "1693312915.384959 ","[{""user"": ""aganivada"", ""timestamp"": ""1693312915.384959"", ""content"": ""<@U03KLHDKL1H> can I close the tenancy HF branch?""}, {""user"": ""askumar"", ""timestamp"": ""1693312939.591819"", ""content"": ""yes <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1693314213.070739"", ""content"": ""<@U03KLHDKL1H> PTAL there were some merge conflicts while merging with develop. Please cross check if all of your changes are present""}, {""user"": ""askumar"", ""timestamp"": ""1693314839.037619"", ""content"": ""Thankyou <@U02BV2DGUKC> validated my changes are there in the merge commit""}]" "1679321938.985919 ","[{""user"": ""aganivada"", ""timestamp"": ""1679321938.985919"", ""content"": ""<@U03KLHDKL1H> did we get a chance to try spring override for transaction manager?""}, {""user"": ""askumar"", ""timestamp"": ""1679377769.187819"", ""content"": ""Not yet...I haven't been able to find a working approach for overriding the context.""}, {""user"": ""aganivada"", ""timestamp"": ""1679378599.401379"", ""content"": ""<@U03KLHDKL1H> qc?""}, {""user"": ""askumar"", ""timestamp"": ""1679378645.498869"", ""content"": ""yes""}]" "1675676945.028849 ","[{""user"": ""aganivada"", ""timestamp"": ""1675676945.028849"", ""content"": ""<@U03DHUAJVMK> /<@U03NZ7Z52S2> with MFA we have a major issue where customer has MFA enabled but vendor does not have MFA, As of today if customer enables MFA then vendor user is not able to access customers product details page because though vendor user has a obfuscated:read role in customer org MFA check if denying access to vendor. I made a change in auth0 rules of int and stage to skip MFA check if the roles is obfuscated:read can you kindly verify the fix in int/stage? we need to make sure though MFA is enabled for customers vendor is still able to access customer data though MFA is not enabled for vendor org.\n\nThis is currently blocking splunk from accessing EA's product details page since EA enabled MFA. Please check and let me know if you have any questions. cc: <@U033PPLNFRU> <@U02GC8SE18V> \n\n\n\n> if(context.authorization && context.authorization.roles){\n> if (context.authorization.roles.includes('obfuscated:read')){\n> console.log(\""User is a obfuscated user, skipping MFA\"");\n> return callback(null, user, context);\n> } \n> }""}, {""user"": ""aganivada"", ""timestamp"": ""1675678352.764739"", ""content"": ""added for this""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1675678624.759119"", ""content"": ""<@U02BV2DGUKC> - quick question - when a user is added to an organization - which has specific policy like MFA ) - what should be ideal behaviour - say everytime when a user from another or is invited to this organization - when he accepts - should we make MFA mandatory for user whenever he comes to this org or skip all securitu setting if it is not his default org ?""}, {""user"": ""aganivada"", ""timestamp"": ""1675678754.446379"", ""content"": ""<@U0336QZAF98> we are only skipping security check of MFA if the role is obfuscated:read which is dedicated to vendors accessing customer data. otherwise all security settings are applicable for any other roles/invitation sent from one org to another""}, {""user"": ""aganivada"", ""timestamp"": ""1675678848.022449"", ""content"": ""BTW this is only a temporary thing since once customers enable SSO all the MFA related settings will be done on SSO""}, {""user"": ""aganivada"", ""timestamp"": ""1675678874.340959"", ""content"": ""we are running into this issue since today we are giving a mock implementation of MFA for customers who don't have SSO yet""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1675679274.663389"", ""content"": ""<@U02BV2DGUKC> - quick question - if customer org has policy of 30 days policy expiry and vendor org has policy of 15 days - which policy will be applied to user ?""}, {""user"": ""aganivada"", ""timestamp"": ""1675679312.944989"", ""content"": ""if the user's primary org is vendor then it will be 15 days""}, {""user"": ""anair"", ""timestamp"": ""1675717597.755649"", ""content"": ""<@U02BV2DGUKC> <@U0336QZAF98> but even with SSO we require vendors to access customer data (and vendors do not need to have SSO)""}, {""user"": ""aganivada"", ""timestamp"": ""1675740931.422139"", ""content"": ""yes <@U033PPLNFRU> this should work even in case of SSO what I meant was with SSO we wont need any custom handling of MFA from auth0 because MFA setting will then be linked to their IDP instead of auth0. Vendor should still be able to access customer data.""}, {""user"": ""aganivada"", ""timestamp"": ""1675748981.078929"", ""content"": ""<@U03NZ7Z52S2> let us know once we are done testing the change will enable it in prod""}, {""user"": ""bganganna"", ""timestamp"": ""1675754332.810009"", ""content"": ""<@U02BV2DGUKC> Only below use-cases are tested :\n1. Customer MFA enabled and vendor has not enabled:\nAble to access product details page from vendor side\n2. Add the collaborator for the customer\nAfter accepting the invite , collaborator needs to enable MFA & for the first time he would be able to access the dashboard. For the next time login , we need to add mfa_enabled:true at the user metadata for login to work .""}, {""user"": ""bganganna"", ""timestamp"": ""1675754349.141779"", ""content"": ""cc <@U03DHUAJVMK>""}, {""user"": ""aganivada"", ""timestamp"": ""1675761710.705279"", ""content"": ""thank you <@U03NZ7Z52S2>""}, {""user"": ""aganivada"", ""timestamp"": ""1675761796.720779"", ""content"": ""there are no changes related to (2) so I think this is existing behaviour. However I think we can live with it because one of the pre-requisites of MFA is to have mfa_enabled:true set for all collaborators""}, {""user"": ""bganganna"", ""timestamp"": ""1675766472.592419"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1675769731.030839"", ""content"": ""<@U040RCBPBEC> <@U028EDANJM9> <@U0431DZTPJM> enabled auth0 rule fix for PLAT-1656 in prod and verified fix, splunk and axiamatic should now be able to access EA's product details. In case we notice any issues in prod related to MFA with new change please disable rule \""*Require MFA Enrollment for access to org - fixes PLAT-1656*\"" and enable back old rule \""*Require MFA Enrollment for access to org*\"" . Unfortunately auth0 rules do not support versioning, will try to move this rule also to action. cc: <@U033PPLNFRU>""}, {""user"": ""aganivada"", ""timestamp"": ""1675769824.304049"", ""content"": ""<@U033PPLNFRU> should we also make similar change to allow access to system_user without MFA?""}]" "1690387130.221609 ","[{""user"": ""aganivada"", ""timestamp"": ""1690387130.221609"", ""content"": ""<@U04JT69T00K> can you check the gitlab runner instance? looks like builds are taking longer than usual cc: <@U0336QZAF98>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690393769.535949"", ""content"": ""Did we resolve this?""}, {""user"": ""aganivada"", ""timestamp"": ""1690451872.077389"", ""content"": ""Cc: <@U04JT69T00K>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690485570.688529"", ""content"": ""<@U04JT69T00K> - did we resolve this? Please close the loop. Thanks""}]" "1685599417.062969 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1685599417.062969"", ""content"": ""Hey <@U02BV2DGUKC> - Should we need other alternative - based on logz support - they have disabled one of the alert bcoz of perf issues in cluster\n```I am contacting you today as we have received several internal notifications that your alert \""[Int] [Global] ERROR logs > 100 in 15mins\"" on account Axiamatic (389045) is causing performance issues on your accounts cluster. \n \nThis alert looks to be using a very heavy search query. To prevent any performance issues we have disbaled the alert in the meantime. We would like to optimize your search query if possible to help improve performance before re-enabling. \n \nWe notice you are using a lot of OR conditions in these alerts. We would recommend making this query more specific or perhaps creating several alerts to cover what this one alert is doing to increase performance. \n \nPlease let us know if you have any questions or concerns. We would also be happy to help optimize your search query further on chat.```""}, {""user"": ""aganivada"", ""timestamp"": ""1685600160.508319"", ""content"": ""<@U0336QZAF98> this is only in int right? I will check and see if we can update the filter""}, {""user"": ""aganivada"", ""timestamp"": ""1685600199.787199"", ""content"": ""Shared a few error cases with teams to see if they can be suppressed I think that should help""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1685600311.701929"", ""content"": ""I think the logs are also getting filled really crazy in INT""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1685600347.163989"", ""content"": ""Saw couple of no storage mails in INT in last two days""}, {""user"": ""aganivada"", ""timestamp"": ""1685600572.570569"", ""content"": ""yeah""}, {""user"": ""aganivada"", ""timestamp"": ""1685600593.728139"", ""content"": ""I also reduced the size of logs in int to provide more space for stage""}, {""user"": ""aganivada"", ""timestamp"": ""1685622097.545299"", ""content"": ""<@U0336QZAF98> turns out we were running out of capacity because of debug logs we enabled last week while debugging a library issue for apps service in int :man-facepalming:. I should have reverted the log level after testing""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685657421.847739"", ""content"": ""<@U02BV2DGUKC> - did you see if the logging stopped after you updated the log level?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685657455.436709"", ""content"": ""I added a tiny bit of capacity to INT to debug something. Bounced onboarding and pulse-survey just in case.""}, {""user"": ""aganivada"", ""timestamp"": ""1685675819.821809"", ""content"": ""<@U026PMDB1ND> for main instances it did stop but we had some multi-instances of app services where log config didnt seem to reflect so I asked Prabhu to stop the instances temporarily but by then we already ran out of little capacity I added. Will take a closer look today now that debug logs have stopped ""}]" "1689244486.577699 ","[{""user"": ""akasim"", ""timestamp"": ""1689244486.577699"", ""content"": ""\nWhen I tried to deploy scoring_service in INT, it failed with the following error:\n```botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the DescribeDBClusters operation: User: arn:aws:iam::433798924509:user/akasim is not authorized to perform: rds:DescribeDBClusters on resource: arn:aws:rds:us-west-2:433798924509:cluster:aslv2-common-int with an explicit deny in an identity-based policy```\nFYI: I was on VPN\nCan someone help here?\ncc: <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1689244517.388019"", ""content"": ""<@U04JT69T00K> can you take a look. it seems this is happening through VPN""}, {""user"": ""askumar"", ""timestamp"": ""1689244581.879869"", ""content"": ""<@U02HQ78V9A5> looks like this permission has not been enabled , this happens when your deployment is trying to call RDS to describe the cluster to which it is going to connect.""}, {""user"": ""aganivada"", ""timestamp"": ""1689244635.145309"", ""content"": ""<@U03KLHDKL1H> do we have to give permissions individually or at group level?""}, {""user"": ""askumar"", ""timestamp"": ""1689244690.152289"", ""content"": ""<@U02BV2DGUKC> this is on group level nobody else faced this issue. <@U04JT69T00K> can tell better.""}, {""user"": ""aganivada"", ""timestamp"": ""1689244731.321699"", ""content"": ""hmm thats weird did we update parameters to rds proxy?""}, {""user"": ""askumar"", ""timestamp"": ""1689244772.229579"", ""content"": ""No <@U02BV2DGUKC> Proxy is not being used""}, {""user"": ""askumar"", ""timestamp"": ""1689244820.004389"", ""content"": ""<@U02HQ78V9A5> I can try the deployment if you want., Prashant is on his way home.""}, {""user"": ""akasim"", ""timestamp"": ""1689244849.185139"", ""content"": ""Sure <@U03KLHDKL1H>\nI will call you""}, {""user"": ""aganivada"", ""timestamp"": ""1689244889.078379"", ""content"": ""<@U03KLHDKL1H> in case if it fails with same error lets add permission to powerusergroup in IAM for now""}, {""user"": ""askumar"", ""timestamp"": ""1689245725.723689"", ""content"": ""<@U02BV2DGUKC> it did not fail with the same error, that describe step passed for me.\n\nNow actually the image which we are deploying is yet to be published, which <@U02HQ78V9A5> is following up.""}, {""user"": ""pjha"", ""timestamp"": ""1689247002.694149"", ""content"": ""<@U02HQ78V9A5> can we connect ?""}, {""user"": ""akasim"", ""timestamp"": ""1689247033.060959"", ""content"": ""sure <@U04JT69T00K>""}, {""user"": ""askumar"", ""timestamp"": ""1689247791.884319"", ""content"": ""Application failing with bean issue after image publishing:\n<@U02HQ78V9A5> <@U04JT69T00K>\n\nException encountered during context initialization - cancelling refresh attempt: org.springframework.boot.context.properties.ConfigurationPropertiesBindException: Error creating bean with name 'druidConfiguration': Could not bind properties to 'DruidConfiguration' : prefix=druid, ignoreInvalidFields=false, ignoreUnknownFields=true; nested exception is org.springframework.boot.context.properties.bind.BindException: Failed to bind properties under 'druid' to com.axm.vms.scoring.provider.config.DruidConfiguration""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689276248.424539"", ""content"": "" - what was the conclusion? Just curious""}, {""user"": ""askumar"", ""timestamp"": ""1689313617.625539"", ""content"": ""<@U026PMDB1ND> the image was deployed successfully by Anz after getting permission.\n<@U04JT69T00K> can you please add what was the group for which Anz needed permission ?""}, {""user"": ""aganivada"", ""timestamp"": ""1689313916.048919"", ""content"": ""but Anzar was alredy part of powerusergroup right <@U03KLHDKL1H>?""}, {""user"": ""askumar"", ""timestamp"": ""1689313941.375519"", ""content"": ""Yes <@U02BV2DGUKC>, not sure what was exact issue.""}, {""user"": ""pjha"", ""timestamp"": ""1689313962.295189"", ""content"": ""I have added RDS-Full access for the testing""}, {""user"": ""pjha"", ""timestamp"": ""1689313971.215279"", ""content"": ""it' worked, I have removed it now""}, {""user"": ""aganivada"", ""timestamp"": ""1689313987.758549"", ""content"": ""<@U04JT69T00K> explicitly to Anzar or to powerusergroup?""}, {""user"": ""pjha"", ""timestamp"": ""1689314010.703479"", ""content"": ""<@U02BV2DGUKC> to Anzar""}, {""user"": ""aganivada"", ""timestamp"": ""1689314061.789869"", ""content"": ""ok then how is it that deploy is succeeding for other services?""}, {""user"": ""aganivada"", ""timestamp"": ""1689314085.851459"", ""content"": ""let me know when you are free <@U04JT69T00K>, lets have a qc""}]" "1683696958.568249 ","[{""user"": ""aganivada"", ""timestamp"": ""1683696958.568249"", ""content"": "" I am suffering with stye trying to reduce screen-time I will be on and off today. Might respond to slack little late.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1683697003.705859"", ""content"": ""Take care <@U02BV2DGUKC>""}, {""user"": ""askumar"", ""timestamp"": ""1683697030.969749"", ""content"": ""Take care Anil...avoid long stretches""}]" "1684215382.904919 ","[{""user"": ""ppant"", ""timestamp"": ""1684215382.904919"", ""content"": ""<@U02BV2DGUKC> I am redeploying the lambda from CDK V2, let\u2019s see if that fixes the issue""}, {""user"": ""aganivada"", ""timestamp"": ""1684215470.359689"", ""content"": ""<@U0431DZTPJM> it should be already cdkv2 ""}]" "1676885034.950599 ","[{""user"": ""askumar"", ""timestamp"": ""1676885034.950599"", ""content"": "" please review :\nApproach to update/fetch stage of epoch execution\n""}, {""user"": ""ppant"", ""timestamp"": ""1676885121.263749"", ""content"": ""Not able to open this""}, {""user"": ""askumar"", ""timestamp"": ""1676885129.691849"", ""content"": ""cc <@U0336QZAF98>""}, {""user"": ""askumar"", ""timestamp"": ""1676885139.003329"", ""content"": ""Added the access <@U0431DZTPJM>""}, {""user"": ""aganivada"", ""timestamp"": ""1676885673.047669"", ""content"": ""<@U03KLHDKL1H> can we also share this in design channel and tag tls for review?""}, {""user"": ""askumar"", ""timestamp"": ""1676885685.713189"", ""content"": ""sure <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1676886213.882889"", ""content"": ""<@U03KLHDKL1H> added some comments""}]" "1679400916.716529 ","[{""user"": ""pjha"", ""timestamp"": ""1679400916.716529"", ""content"": "" Here is the confluence design doc for the \""Deploying multiple instances of same service\"" ""}, {""user"": ""aganivada"", ""timestamp"": ""1679559532.744979"", ""content"": ""<@U04JT69T00K> can we post this in design-review channel?""}, {""user"": ""pjha"", ""timestamp"": ""1679559551.287689"", ""content"": ""sure""}]" "1686249255.904479 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1686249255.904479"", ""content"": "" <@U02BV2DGUKC> <@U028EDANJM9> <@U02HCMTQU3W> - In logz for integration account , all the fields under mdc are getting dropped and not indexed . When i checked with their support - It looks like the field \u201cmdc.tenantId\u201d is mapped as a long but was sent with a null value- This is coming as part of event framework - SnsEventPublisherImpl.java - I have asked the support to create a rule - where it will accept only if it comes as long instead of string\n```In our log files - we see this line - logzio_removed_fields\nThe following fields were removed due to conflicting types:\nfargate -> mdc: {batch_id=3f737948-840c-8054-b1b1-a564740a71c9, index=2, productInstanceId=35619, pulseInstanceId=41654, questionsCount=0, tenantId=51165, tracking_id=07f80e2c-3b60-4d5f-9689-660400a5df9c}```""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1686249330.053829"", ""content"": ""Here is the shortcut to view logs which are having problem\n""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1686249365.923499"", ""content"": ""Lucene Query : type:logzio-index-failure AND index-failed-reason:/.*mdc.tenantId.*/""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686249392.922999"", ""content"": ""Are they all from the discovery service?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1686249501.054929"", ""content"": ""i see \u201cUserAttributeDiscoveryPublisher.java\u201d in all the errors - i think this is the new feature""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1686249504.574769"", ""content"": ""in discovery""}, {""user"": ""gshenoy"", ""timestamp"": ""1686282819.867899"", ""content"": ""Thanks <@U0336QZAF98>, the delegate had the annotation `EnableMDCPublicMethods` but the tenantId field is optional.\nI have removed the annotation now.""}, {""user"": ""aganivada"", ""timestamp"": ""1686282919.642699"", ""content"": ""thank you <@U0336QZAF98> is this the same issue logz sent us a note about last night?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1686283039.830149"", ""content"": ""Yes <@U02BV2DGUKC> - right now they have added a rule - if in json tenantId field comes as string - they map to tenantId_str - if it comes as int - it goes to tenantId""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1686283099.026739"", ""content"": ""The problem is - even if it is a valid number - but it comes as say '45'- it is going to _str field""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1686283116.411489"", ""content"": ""as it is treated as json string""}, {""user"": ""aganivada"", ""timestamp"": ""1686283158.799039"", ""content"": ""ok this might break some filters and alerts, probably we could set an alert where _str is present it means the code needs to fix it""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1686283226.667399"", ""content"": ""If fixes from discovery team goes in , then we can ask to revert this rule""}, {""user"": ""gshenoy"", ""timestamp"": ""1686283271.932759"", ""content"": ""Will deploy it in sometime <@U0336QZAF98>, the build is in progress.""}, {""user"": ""gshenoy"", ""timestamp"": ""1686283316.708569"", ""content"": ""Also, in the common lib for mdc , we may need to add a null check.\n\n\n```final String argNameFromMethod = names[i];\nif (Arrays.stream(argNames).anyMatch(argName -> argName.equals(argNameFromMethod))) {\n ThreadContext.put(argNameFromMethod, String.valueOf(args[i]));\n}```\nCc <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1686283375.045349"", ""content"": ""<@U028EDANJM9> can you add a bug on this? will fix it along with some other bugs Seshan found""}, {""user"": ""gshenoy"", ""timestamp"": ""1686283509.149369"", ""content"": "" <@U02BV2DGUKC>""}]" "1681877334.429929 ","[{""user"": ""aganivada"", ""timestamp"": ""1681877334.429929"", ""content"": ""<@U033PPLNFRU> /<@U04EV0M2VA6> we are working on re-naming the user roles according to , based on product admin and account admin are to be renamed as \""Local Admin\"" I see a open thread in the article on this topic did we finalize on approach? reason is we will have to run a migration script to update all existing product/account admin's to local admin role and also check any open invitations with these roles and re-invite as local admin. Once we run the migration script to change to local admin we cannot revert back the roles. cc: <@U028EDANJM9> <@U02GC8SE18V>""}, {""user"": ""anair"", ""timestamp"": ""1682388032.978719"", ""content"": ""hi <@U02BV2DGUKC> sorry this thread got lost for me. Can we consider the following\n\nOrg Admin --> Enterprise Admin\n\nAccount Admin --> remains the same\n\nProduct Admin --> Vendor Admin""}, {""user"": ""rvaidya"", ""timestamp"": ""1682390137.164139"", ""content"": ""<@U033PPLNFRU> Do we want to differentiate the Product Admin with Account Admin as the privileges are similar for both the roles (with different tenant type) ?""}]" "1677681912.380929 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1677681912.380929"", ""content"": ""<@U033PPLNFRU> <@U040RCBPBEC> <@U02Q02G09QX> - On our retro feedback comments - i have generated excel - containing topics extracted from open text comments for aws comprehend vs openapi_curie vs openapi_davinci - could you pass your feedback on quality of topics""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677697245.909829"", ""content"": ""This is very advanced :joy:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677697295.553919"", ""content"": ""Curie decided that \""on-call\"" is from the medical domain and decided to add something of its own :slightly_smiling_face:""}, {""user"": ""aganivada"", ""timestamp"": ""1677729783.618799"", ""content"": ""<@U0336QZAF98> can't find the file""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1677729857.371109"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1677731933.105259"", ""content"": ""I am not an expert but curie seems to have done a decent job, few of observations:\n\n1. aws_topics shows empty for a lot of cases also on average topics dont seem to covey important information\n2. In some cases when there are bullet points Curie seems to extract the entire string is this expected?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1677739385.349259"", ""content"": ""<@U02BV2DGUKC> - yes - i am investigating this""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1677772568.086789"", ""content"": ""OpenAI has released gpt-3.5-turbo which costs almost same as curie and 1/10 price of davinci . I ran our retro comments with new model - keyword extraction is looking way better - This time for sentiment calculation - we gave question also along with opentext answer to give much better context - let me know your thoughts <@U033PPLNFRU> <@U040RCBPBEC> <@U026PMDB1ND> <@U02BV2DGUKC>""}, {""user"": ""svummidi"", ""timestamp"": ""1677773857.052369"", ""content"": ""<@U0336QZAF98> - For now we don\u2019t have question in the pipeline, we need to work with <@U02SF36PVKL> to get it part of the ingestion flow.\nEven <@U02Q02G09QX>\u2019s experiments also proves that adding question context helps for openAI""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1677775967.803489"", ""content"": ""sure <@U040RCBPBEC>. Here is the repo - where i have checked in the script - - for prompts - please review\nwill check with <@U02SF36PVKL> on questions""}, {""user"": ""svummidi"", ""timestamp"": ""1677776160.781759"", ""content"": ""<@U0336QZAF98> It looks like there is no concept of session.\nAs we discussed let us collect the tokens consumed count for better comparison. With turbo they eliminated config parameters but it increases the token count. Not a major concern but it is better to track that for easy comparison between models and prompts.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1677776363.911449"", ""content"": ""sure <@U040RCBPBEC> .will update the script to record tokens for comparison""}]" "1685419995.607239 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1685419995.607239"", ""content"": ""<@U02BV2DGUKC> <@U04JT69T00K> - take a quick look at this: \n\nParticularly the very last file in the list. I am just adding a few more default permissions to ECS tasks to add SSM params""}, {""user"": ""pjha"", ""timestamp"": ""1685423686.506979"", ""content"": ""<@U026PMDB1ND> LGTM""}, {""user"": ""aganivada"", ""timestamp"": ""1685425517.857589"", ""content"": ""<@U026PMDB1ND> for permissions would it make sense to add some additional prefix to the SSM parameters?""}, {""user"": ""aganivada"", ""timestamp"": ""1685425552.090259"", ""content"": ""since all params have env variable as prefix it might be hard tracking the ones added only for admin-gw""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685425617.510319"", ""content"": ""Not really, <@U02BV2DGUKC>. There\u2019s no harm in letting all services in a env have read access to parameters in that env. I was surprised that we didn\u2019t add it till now.""}, {""user"": ""aganivada"", ""timestamp"": ""1685425634.532059"", ""content"": ""sure Rama""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685425813.924479"", ""content"": ""So far we have been picking them up from\nssm and setting them in the env during provisioning but it is also useful to let the service read what it needs directly (a bit like cloud config). ""}, {""user"": ""aganivada"", ""timestamp"": ""1685426069.253179"", ""content"": ""ok, I think we might have a few cases where we read ssm from code may be we added explicit permissions on those keys""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685426093.342369"", ""content"": ""Possible""}]" "1684746143.559989 ","[{""user"": ""sranjan"", ""timestamp"": ""1684746143.559989"", ""content"": "" <@U02BV2DGUKC> <@U03KLHDKL1H> We are getting so many log as \""*Failed to parse given rule 17207::c::17207:pin:21772::rw . so skipping the rule*\"" IN *STAGE .*""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1684747815.994399"", ""content"": ""<@U034RLJA97X> - For which component - this is happening ?""}, {""user"": ""sranjan"", ""timestamp"": ""1684747865.235169"", ""content"": ""Jira container""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1684748016.110599"", ""content"": ""<@U034RLJA97X> what is the version of platform libs common in your pom file ?""}, {""user"": ""sranjan"", ""timestamp"": ""1684748317.852799"", ""content"": ""<@U0336QZAF98> axm-commons.version : *0.0.10*""}, {""user"": ""aganivada"", ""timestamp"": ""1684748349.117009"", ""content"": ""Seems quite old""}, {""user"": ""aganivada"", ""timestamp"": ""1684748359.787199"", ""content"": ""<@U034RLJA97X> can we update to latest?""}, {""user"": ""aganivada"", ""timestamp"": ""1684748389.626789"", ""content"": ""At service level though this error may be harmless unless you are using rbac in jira service""}, {""user"": ""sranjan"", ""timestamp"": ""1684748447.718209"", ""content"": ""<@U02BV2DGUKC> It was flooding the logz , so raised .""}, {""user"": ""aganivada"", ""timestamp"": ""1684748572.973149"", ""content"": ""Ok we did enable a flag to use new permission model may be that is why we are seeing this we can check once you update the commons version. However, it should happen only for UI calls.""}, {""user"": ""sranjan"", ""timestamp"": ""1684748656.935749"", ""content"": ""Thanks <@U0336QZAF98> <@U02BV2DGUKC>. Will update axm-version . Created a tracker ticket.""}]" "1675224541.007139 ","[{""user"": ""aganivada"", ""timestamp"": ""1675224541.007139"", ""content"": "" <@U03DHUAJVMK> <@U03NZ7Z52S2> <@U03KLHDKL1H> <@U033PPLNFRU> reviewed bugs added in 0.9.7 for collaborators as fix-in label is not added yet \n\n1. PLAT-1642 - <@U03KLHDKL1H> can you check with <@U02GC8SE18V> on this? it seems somehow permissions are not being verified properly for ProductInstanceServiceHelper > updateProductInstance. \n2. PLAT-1641 - need some info on onboarding flow it seems for some reason permissions are not set during onboarding flow. Verified some other calls where permissions were set <@U03KLHDKL1H> please take a look once we have info from Anu on this\n3. PLAT-1640 - should be straightforward updating error message from backend, we can fix it once fix-in label is added cc: <@U02HCMTQU3W> \n4. - need more info on this. tried to login to integration as user but got prompted for MFA\n5. PLAT-1636 & 1637 - will require UI and backend changes, we can fix this as HF or next release.\n""}, {""user"": ""askumar"", ""timestamp"": ""1675225195.324289"", ""content"": ""Looking into the assigned bugs""}, {""user"": ""aganivada"", ""timestamp"": ""1675227963.333939"", ""content"": ""<@U03KLHDKL1H> JFYI PLAT-1641 is closed after discussing with <@U03DHUAJVMK>""}, {""user"": ""askumar"", ""timestamp"": ""1675247540.423559"", ""content"": ""For PLAT-1642\n\nHere the update API has authorizer that is checking for condition that user shoukd be orgAdmin and have permissions as well.\n @PreAuthorize(\""@axmAuthorizer.isOrgAdmin() \""\n + \""&& @axmAuthorizer.allowedProductInstanceV2(#productInstanceUpdateRequestDTO.getId(), #tenantId, 'readWrite')\"")\nThis is an AND condition that is denying update and limiting the update only for orgAdmin.\n""}, {""user"": ""rvaidya"", ""timestamp"": ""1675248234.409179"", ""content"": ""<@U033PPLNFRU> <@U03DHUAJVMK> PLAT-1642 since priority is feature flagged and not enabled for any customer, and as you mentioned on the jira that this can slip ..we wont HF this.\n\nHowever we should fix it in develop anyways <@U03KLHDKL1H>.\n\ncc <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1675254708.076409"", ""content"": ""<@U03KLHDKL1H> so the fix is changing it to OR instead of AND ? Not sure how this slipped, current condition doesn't make sense. Thank you for debugging this <@U02GC8SE18V> & <@U03KLHDKL1H>""}, {""user"": ""askumar"", ""timestamp"": ""1675254776.443719"", ""content"": ""yes <@U02BV2DGUKC>..the same OR condition is there in deleteProduct as well""}]" "1682336186.585829 ","[{""user"": ""aganivada"", ""timestamp"": ""1682336186.585829"", ""content"": ""<@U026PMDB1ND> me,<@U03KLHDKL1H> and <@U04JT69T00K> had a discussion regarding the DB migration to aurora serverless v2 in integration. Based on current setup we have 3 rds instances with current config we might be charged ~ USD 81 per instance type. With aurora serverless v2 we might be charged 131 with proxy and 43 without proxy (Not sure which mode we choose for prod). Please review the cost calculator \n\nDuring our discussion we came up with following options:\n1. Migrate 3 rds instances to 2 aurora serverless and remove all existing rds instances: commons => commons + discovery, core => core\n2. Migrate just core and discovery to aurora instance and remove existing rds instances for core and discovery, leave commons the way it is currently (RDS postgres)\nwe are thinking (2) might reduce price (assuming we go with ) and downtime on integration as we will not update existing instances but it may introduce some inconsistencies with some services using aurora and some using regular postgres.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682395910.825369"", ""content"": ""<@U02BV2DGUKC> - I think we should just go with Aurora for all since it would mirror what we are doing in prod.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682395925.807109"", ""content"": ""I don't think we are doing proxy in prod""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682395985.877139"", ""content"": ""If we use RO/RW we can get much better value from Aurora""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682396074.128529"", ""content"": ""We can try proxy in INT and if it works out well we could introduce it in prod as well.""}, {""user"": ""aganivada"", ""timestamp"": ""1682396881.498779"", ""content"": ""sure thank you <@U026PMDB1ND>, we will also use same config (aurora for all instances) in stage as well""}]" "1679292204.788369 ","[{""user"": ""rvaidya"", ""timestamp"": ""1679292204.788369"", ""content"": ""<@U02BV2DGUKC> In INT, tenancy is generating lot of logs like\nentering, exiting each method call etc. Is it intentional? Are you debugging something?""}, {""user"": ""aganivada"", ""timestamp"": ""1679292563.692349"", ""content"": ""checking <@U02GC8SE18V> , is it for all tenants or just 19190? I dont see any log config changes in last 2 weeks ""}, {""user"": ""rvaidya"", ""timestamp"": ""1679292672.435449"", ""content"": ""Trace id : 6417f454b2feb22cb132fe4fcad726e0""}, {""user"": ""aganivada"", ""timestamp"": ""1679292754.261099"", ""content"": ""seems like trace is enabled for tenantid 19190, will disable it cc: <@U0431DZTPJM>""}, {""user"": ""aganivada"", ""timestamp"": ""1679292817.144119"", ""content"": ""<@U0431DZTPJM>, can we check why we are printing trace logs when only debug is enabled for tenant level filter?\n\n> ``` <Filters>\n> <AxmFilter maxApplicationLevel=\""debug\"" tenants=\""19190\""/>\n> </Filters>```\n> \n""}, {""user"": ""aganivada"", ""timestamp"": ""1679389622.106989"", ""content"": ""<@U0431DZTPJM> some service seems to be excessively logging can we check in logz ? we can filter on debug logging from int and stage to check if log config changed for any service""}, {""user"": ""ppant"", ""timestamp"": ""1679389908.021439"", ""content"": ""<@U02BV2DGUKC> I looked at the stage and dev configs. In both int and stage only scheduler has axm filter but no tenants mentioned\nstage conf - \nint conf - ""}, {""user"": ""aganivada"", ""timestamp"": ""1679389936.553129"", ""content"": ""ok""}, {""user"": ""ppant"", ""timestamp"": ""1679390048.040749"", ""content"": ""ingestion service seems to be producing lots of debug logs\n""}, {""user"": ""aganivada"", ""timestamp"": ""1679390051.801059"", ""content"": ""<@U0336QZAF98> ingestion service seems to have debug enabled, is this expected? ""}, {""user"": ""aganivada"", ""timestamp"": ""1679390053.686419"", ""content"": ""yeah""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1679390088.053819"", ""content"": ""No <@U02BV2DGUKC> - let me check""}, {""user"": ""ppant"", ""timestamp"": ""1679392639.089969"", ""content"": ""<@U02BV2DGUKC> sentinment collector has the max logs in last 5000 occurences in int""}, {""user"": ""aganivada"", ""timestamp"": ""1679392734.715749"", ""content"": ""<@U03RSS0S76Y> can we revert log level to info? ""}, {""user"": ""snangia"", ""timestamp"": ""1679392854.615129"", ""content"": ""<@U02BV2DGUKC> will do it tomorrow.""}, {""user"": ""svummidi"", ""timestamp"": ""1679437969.438769"", ""content"": ""<@U0336QZAF98> - I noticed debug enabled at spring level for ingestion service, updated the xml file but not sure if I restarted the service. Is there any other issue related to logging?\n""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1679459140.196299"", ""content"": ""<@U040RCBPBEC> - looks like auto refresh of log level when changed via cloud config is not reflecting - at the end I ended up bouncing""}, {""user"": ""ppant"", ""timestamp"": ""1679459937.698899"", ""content"": ""<@U0336QZAF98> We faced a similar issue with core data few weeks back, it\u2019s a known bug with Log4j2 ""}, {""user"": ""aganivada"", ""timestamp"": ""1679637596.701209"", ""content"": ""<@U03RSS0S76Y> can we update SC logging level back to info? if it is updated but log config changes is not reflecting then please bounce service in int""}, {""user"": ""snangia"", ""timestamp"": ""1679638335.153689"", ""content"": ""yes <@U02BV2DGUKC> just completed final testing, will do so.""}]" "1692942863.174719 ","[{""user"": ""aganivada"", ""timestamp"": ""1692942863.174719"", ""content"": ""<@U04JT69T00K> in the 2 highlighted instances maximum acu is 5 and 3 so we should be good right? Would minimum acu cause any issues?""}, {""user"": ""pjha"", ""timestamp"": ""1692943624.525009"", ""content"": ""<@U02BV2DGUKC> According to the support, During the first attempt to upgrade, the minimum ACU was 0.5 and maximum ACU was 1. On reviewing the ServerlessDatabaseCapacity, we can see that the ACUs at the time of upgrade was 0.5 which means it had 1 GiB memory available. From the available 1 GiB memory, the Freeable Memory was around 300 MB.\n\nWhen I inquired the configuration where the minimum ACU is 0.5 and the maximum ACU is 1, I was anticipating having a 2 GiB of available memory, their response clarified that having a maximum ACU of 1 does not ensure a constant 2 GiB of memory. Instead, the available memory fluctuates between 1 GiB and 2 GiB in accordance with the scaling of ACUs on the cluster. The memory scales in line with the changes in ACUs.""}, {""user"": ""pjha"", ""timestamp"": ""1692943751.976179"", ""content"": ""I think we should increase the minimum ACU to be on the safer side.""}]" "1678865294.433639 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1678865294.433639"", ""content"": ""<@U03ML44G5RC> - by any chance did you update the changeset in core after deploying it once?\n\nException encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [org/springframework/boot/autoconfigure/liquibase/LiquibaseAutoConfiguration$LiquibaseConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.ValidationFailedException: Validation Failed:\n 1 change sets check sum\n db-patches/changelog/include/db.changelog-1.3.xml::svc-core-data-DISC-2609-01:: was: 8:1c7c47bcb3ee1ff82290c83ddc10171e but is now: 8:f8cdd59df18847bb600377a37400515f""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678865788.498489"", ""content"": ""Should I drop the persona table and remove the changelog?""}, {""user"": ""sbhosale"", ""timestamp"": ""1678865808.400299"", ""content"": ""<@U026PMDB1ND>\nI did add an unique constraint but accidentally it went as part of another mr..""}, {""user"": ""sbhosale"", ""timestamp"": ""1678865831.490179"", ""content"": ""yes please do that if possible or i can add it as different change set.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678865911.480789"", ""content"": ""Let me try""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678866477.405499"", ""content"": ""<@U03ML44G5RC> - Now the persona table is same as what is in develop's svc-core-data-DISC-2609-01 changeset.\n\nIf needed, please just add another changeset to modify the table""}, {""user"": ""sbhosale"", ""timestamp"": ""1678866616.185109"", ""content"": ""Thanks <@U026PMDB1ND> will do that.""}]" "1683142821.777179 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1683142821.777179"", ""content"": "" - the ~upcoming~ current platform sprint is the 100th platform sprint :slightly_smiling_face:""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1683201349.855949"", ""content"": ""awesome""}]" "1689362315.020609 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1689362315.020609"", ""content"": ""<@U0336QZAF98> - Looking at notification service to see how it uses SQS. I have a question re: the use of @Async when it comes to REST -> SQS transition. Why would we want to ack the sender before a message is in the queue? Is there a chance that we accept a bunch of messages and queue it for the async executor, but meanwhile if there's a node death, the client would assume that the messages went out though they didn't? Would we lose anything if we submitted the messages to SQS in a sync fashion and then responded to the caller so that we can guarantee a no-loss handoff?\n\nIt is still possible that even in the sync mode, after submitting N-1 of N messages if the notification node dies, the caller would get an error and it would replay and possibly cause a dups. At least there would be no drops.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689362345.888469"", ""content"": ""Just trying to understand if it was done for a specific purpose.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689366380.294799"", ""content"": ""<@U026PMDB1ND> - The main usecase was - via sendemail api - you can send email to multiple recipients bulk and we create tracking id for each one of recipient - if we do a sync call to sqs - if I have 10 recipients - each recipient will have one sqs message - there will be a latency if call bulk sqs post or send msg one by one in sync fashion - that's why this endpoint is async - For that purpose we have a tracking table - which maintains tracking id in postgres db with status as created till it gets posted to sqs - if there is failure node crash - all these messages will be in status created in postgres db and never posted to sqs - I wanted to handle this use case - where - at the start of service - if there are any created state emails in postgres , send to sqs as part of service startup - but it did not materialize bcoz of spam issues and trying other smpt providers""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689371183.269299"", ""content"": ""Oh okay.. so we do have a store which gets populated before return from the REST call. I looked at the tracking table but it doesn't seem to carry the content. Is the content being saved somewhere else if we need to replay the \""created\"" e-mails on restart? If we have to write the content somewhere, we could write it to SQS itself, right?\n\nI wonder if we'd ever get such a huge batch of e-mails that we can't write to SQS in short order. If there's a user involved it is a bit different, but if it is service to service, maybe can be a bit slow (in this case, being certain that the mail has been stored somewhere is better).\n\nOn a side note, I think we need to write some code to clean up the email-tracking table. We should probably clean up once a day all the published e-mails > 30 days and all the created e-mails > 60days (<@U0336QZAF98> was this something we wanted to do?)\n\nIn prod db, we have e-mails from Aug 2022. 20k in created state, 25k in published state and 14 in failed state. There are 3.8k CREATED in the last 1 month. Is there a known reason why they'd be in that state? A few I eyeballed were all going to @onmicrosoft accounts.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689371271.121489"", ""content"": ""```postgresdbproduction=> select count(*), email from email_tracking where eid = 'CREATED' and stored_at > '2023-06-15' group by email order by count(*) desc;\n count | email\n-------+----------------------------------------------------------------------\n 300 | \n 297 | \n 291 | \n 289 | \n 287 | \n 286 | \n 285 | \n 256 | \n 244 | \n 235 | \n 209 | \n 196 | \n 196 | \n 45 | \n 38 | \n 24 | \n 21 | \n 21 | \n 21 | \n 19 | \n 19 | \n 19 | \n 19 | \n 19 | \n 10 | \n...\n....```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689371963.934409"", ""content"": ""Do we not send to addresses intentionally?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689374235.017459"", ""content"": ""<@U026PMDB1ND> - This maintains lifecycle of tracking - same tracking_id will have two records one for created and one for published | failed\n```-- eid can be created , published | failed \n\nwith a as (\nselect tracking_id , \n\tmax(eid) as status from email_tracking group by 1\n)\nselect status , count(1) from a group by 1```""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689374485.134029"", ""content"": ""For above query in prod - we have - which means total 14 have created->failed lifecycle and almost 20756 in created->published lifecycle""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689374618.782639"", ""content"": ""there are also 20k in created state, right?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689374647.423879"", ""content"": ""if we take tracking id in one of created state - it will have another record with published state""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689374661.048889"", ""content"": ""Oh okay""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689374674.002719"", ""content"": ""We don't update the state.. Sorry for the confusion""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689374685.904369"", ""content"": ""This is to track how long it took from creation to publish or failed .""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689374701.275019"", ""content"": ""ok""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689374720.247839"", ""content"": ""I guess we should clean up the old records though""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689374777.840219"", ""content"": ""1. For clean up , there is a backlong - we need to go via either pgcron (scheduler inside postgres) or via lambda - There is a backlog ifor this ticket ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689374784.096199"", ""content"": ""Also, couldn't we have added a couple of fields (final state and final timestamp) and just updated the record instead of creating a new one?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689374991.600409"", ""content"": ""For each state - we needed a timestamp - right now we had only only created , published - There was a plan - where we will subscribe to aws ses ( previously sendgrid ) to track whether email is successfully sent - That will be other states - \u201cSENT\u201d , \u201cEmail Clicked\u201d- There is backlog for this - right now published - is we submitted request to ses - it does not mean mail has reached - As we have different lifecycle of states , right tracking table is always in append mode""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689375005.236279"", ""content"": ""Btw, this clean up thing was a tangent. My main question is regarding how we handle a potential failure if it happens right after returning success to the REST client.\n\nAlso, none of this is urgent. Just asked the questions that popped up while reading code.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689375355.672029"", ""content"": ""Agreed Rama - Async was introduced maily for bulk send usecases - but having said that - as you called out - there is potential case where failures can result in lost emails - if we can handle this tradeoff - bulk send vs fast response in efficient way- it will be better""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689375790.610529"", ""content"": "" - This page tracks completed vs pending features""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689380659.387859"", ""content"": ""Thanks <@U0336QZAF98> . I didn\u2019t mean to discuss this during your night though.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689380815.466839"", ""content"": ""Re: handling bulk send, we should just allow the caller to point us to a s3 folder and we can use it as the source. Agree that if we want to support a use case of sending 100s of emails in one shot REST based request response is not ideal.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689380876.970099"", ""content"": ""<@U02GC8SE18V> do you know what is the biggest bulk email send we do now?""}, {""user"": ""rvaidya"", ""timestamp"": ""1689384515.301879"", ""content"": ""<@U026PMDB1ND> it has to be the executive reports. We tried with 1000 comments. We did not have issue with notification service email.\n<@U02SCRTM2M7> <@U03NZ7Z52S2> what's the exact size?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689386405.846239"", ""content"": ""<@U02GC8SE18V> - I mean what is the max users we send to at a time? Size of the email is not that critical for this discussion.""}, {""user"": ""rvaidya"", ""timestamp"": ""1689391453.345049"", ""content"": ""Oh I understood it as size. We have provision to send to multiple users, but I don't have a number. May be we need to check the api logs or put additional logging for the count of users it is getting sent in prod. Do u want to limit to some number for sending to bulk users ?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689403050.256699"", ""content"": ""Just trying to understand what is the maximum possible number so that we can decide the next steps. No changes required on the calling side at this point (including logs).""}, {""user"": ""sfarooqh"", ""timestamp"": ""1689574799.588679"", ""content"": ""<@U026PMDB1ND> the bulk limit is 10 users configured in notification service. But from reporting service we are invoking notification-service for 1 user at a time currently. cc: <@U0336QZAF98> <@U02GC8SE18V>""}]" "1684944677.070379 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1684944677.070379"", ""content"": ""<@U03DHUAJVMK> how about the \""previous\"" page?""}, {""user"": ""araman"", ""timestamp"": ""1684945128.858529"", ""content"": """"}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684948245.028029"", ""content"": ""<@U02D4DUKDQC> - is back button not an option in a slack bot? Don't remember if I saw it in Polly either.""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1684949452.930019"", ""content"": ""We had an issue because of follow up questions <@U026PMDB1ND> because the navigation becomes dynamic, but <@U02SF36PVKL> has changed that behaviour where follow up is shown in the same page. <@U02SF36PVKL> Can we review to see if we can introduce the back button line in teams?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684949854.586789"", ""content"": ""<@U02D4DUKDQC> - I don't want to create a new requirement if PM doesn't see one. Just asking to see if this is something we tried and had technical challenges with it""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1684950031.454569"", ""content"": ""yes <@U026PMDB1ND> we had it in original design introduction of followups created complexity and we got rid of it""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684950536.526619"", ""content"": ""oh ok""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684950600.107489"", ""content"": ""I think that users are going to find it very challenging without a back button though I understand the challenges""}]" "1682400318.581139 ","[{""user"": ""pjha"", ""timestamp"": ""1682400318.581139"", ""content"": ""<@U02BV2DGUKC> once we merge this change have to delete and deploy the tenancy.""}, {""user"": ""aganivada"", ""timestamp"": ""1682401572.678009"", ""content"": ""<@U04JT69T00K> can we try deploying tenancy with this change instead of destroying -> deploy?""}, {""user"": ""pjha"", ""timestamp"": ""1682401621.493449"", ""content"": ""since there is a change in the service-id we have to delete it .""}, {""user"": ""aganivada"", ""timestamp"": ""1682401629.013329"", ""content"": ""ok""}, {""user"": ""pjha"", ""timestamp"": ""1682401934.183279"", ""content"": ""I think we also need merge request for 'issue/shortning_service_id' to 'release/0.9.10. ' should I create one ?""}, {""user"": ""aganivada"", ""timestamp"": ""1682403847.291269"", ""content"": ""<@U04JT69T00K> would shortning_service_id have impact at other places as well or just the lb - route53 integration?""}, {""user"": ""pjha"", ""timestamp"": ""1682403962.161179"", ""content"": ""only for loadbalancer - route53 integration .""}, {""user"": ""aganivada"", ""timestamp"": ""1682404478.447039"", ""content"": ""ok cool then we can merge to \""release/0.9.10\"" that will take it to develop and main""}, {""user"": ""pjha"", ""timestamp"": ""1682406363.245179"", ""content"": ""sure""}]" "1685365174.355399 ","[{""user"": ""pjha"", ""timestamp"": ""1685365174.355399"", ""content"": ""<@U02BV2DGUKC> <@U03KLHDKL1H> <@U026PMDB1ND> please review the document to upgrade RDS-Aurora-Postgres engine version to 15.2\n\nhere we have two strategy, please suggest the one .""}, {""user"": ""aganivada"", ""timestamp"": ""1685365241.578489"", ""content"": ""<@U04JT69T00K> are there any AWS recommended strategies? Sorry forgot to mention this earlier during our call""}, {""user"": ""pjha"", ""timestamp"": ""1685365339.972019"", ""content"": ""Let me check""}, {""user"": ""pjha"", ""timestamp"": ""1685366176.619379"", ""content"": ""<@U02BV2DGUKC> AWS recommends the 2nd strategy with few checks prior to upgrading the engine version""}, {""user"": ""pjha"", ""timestamp"": ""1685366180.816389"", ""content"": """"}, {""user"": ""pjha"", ""timestamp"": ""1685366260.423549"", ""content"": ""I think it's looks more natural as upgrade option is provided by AWS""}, {""user"": ""askumar"", ""timestamp"": ""1685366449.292369"", ""content"": ""Does restoring the snapshot also reverts the version <@U04JT69T00K> automatically?""}, {""user"": ""pjha"", ""timestamp"": ""1685366625.840319"", ""content"": ""<@U03KLHDKL1H> it won't""}, {""user"": ""aganivada"", ""timestamp"": ""1685366666.605369"", ""content"": ""<@U04JT69T00K> <@U03KLHDKL1H> can we review the before upgrade checklist? ""}, {""user"": ""aganivada"", ""timestamp"": ""1685368097.539259"", ""content"": ""<@U04JT69T00K> even in aws recommended steps they talk about dry run\n\n> We highly recommend testing a major version upgrade on a duplicate of your production database before trying the upgrade on your production database. You can monitor the execution plans on the duplicate test instance for any possible execution plan regressions and to evaluate its performance. To create a duplicate test instance, you can either restore your database from a recent snapshot or clone your database. For more information, see or .\n\nI can think of 2 approaches\n\nApproach-1:\n1. Clone a recent snapshot - new db instance from snapshot\n2. upgrade the snapshot\n3. connect core-data to new instance make sure everything is working ok\n4. drop the old instance\n5. connect all services to point to new instance\nApproach-2:\n1. Clone a recent snapshot - new db instance from snapshot\n2. Upgrade the active instance with some planned downtime\n3. if we run into any issue then update the mappings to point to new instance\n4. if there are no issues found drop the new instance\ncc: <@U026PMDB1ND>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685407460.972089"", ""content"": ""<@U02BV2DGUKC> <@U03KLHDKL1H> - did we stop any RDS instance in INT? Have we updated the CDK code to point to new instance?\n\nI am trying to deploy a new version of admin-gw and it is not able to connect to the previous core instance (ri1d3dczhul2thv).\n\nI just brought it up for the sake of testing.""}, {""user"": ""aganivada"", ""timestamp"": ""1685417078.527949"", ""content"": ""yes <@U026PMDB1ND> \n\n<@U03KLHDKL1H> submitted MR with CDK changes we are currently checking for regression issues for regular services, we should be able to merge the change by EOD to develop.\n\n> I am trying to deploy a new version of admin-gw and it is not able to connect to the previous core instance (ri1d3dczhul2thv).\n> I just brought it up for the sake of testing.\nsure Rama, once the changes are merged we can make CDK changes on admin-gw to point to the new instance cc: Ashwani""}, {""user"": ""askumar"", ""timestamp"": ""1685417841.877469"", ""content"": ""hey <@U026PMDB1ND> we can just restart the RDS core instance if you need it""}, {""user"": ""askumar"", ""timestamp"": ""1685417853.430889"", ""content"": ""we have not yet deleted it""}, {""user"": ""askumar"", ""timestamp"": ""1685417991.459249"", ""content"": ""Just saw ,you started it :slightly_smiling_face:""}, {""user"": ""askumar"", ""timestamp"": ""1685424424.264609"", ""content"": ""<@U02BV2DGUKC>\nMe and <@U04JT69T00K> ran the given set of commands for Precheck on the clone created with snapshot.\nThe mentioned queries got executed with expected response, so we should be good according to pre requisites.""}, {""user"": ""askumar"", ""timestamp"": ""1685424541.887789"", ""content"": ""cc <@U026PMDB1ND>""}, {""user"": ""aganivada"", ""timestamp"": ""1685425296.290679"", ""content"": ""cool thank you <@U03KLHDKL1H>\n\n<@U026PMDB1ND> for integration we thought of taking approach-1 where we upgrade the snapshot instance and test it by connecting to core, this approach will require updating SSM parameters and deploying services again but if there are any major issues we should be able to identify and switch back to original instance. For stage and prod we can use approach-2 where we keep a cloned instance ready in case there are any issues and upgrade the main instance with some downtime, with approach-2 we dont have to update any SSM parameters as service will continue to use the same db host.""}, {""user"": ""pjha"", ""timestamp"": ""1685430546.906949"", ""content"": ""FYI we are upgrading the RDS instance launched through the snapshot, this won't impact any service. Will update once it's upgraded and we connect core to this .""}, {""user"": ""askumar"", ""timestamp"": ""1685430645.891159"", ""content"": ""cc <@U02BV2DGUKC> <@U026PMDB1ND>""}, {""user"": ""pjha"", ""timestamp"": ""1685442047.912629"", ""content"": ""<@U02BV2DGUKC> <@U03KLHDKL1H> FYI Int common database version upgrade has stated.""}, {""user"": ""pjha"", ""timestamp"": ""1685445038.745899"", ""content"": ""we have upgraded the version and deployed core-data service pointing to the upgraded postgres rds\n\n""}, {""user"": ""aganivada"", ""timestamp"": ""1685445166.621819"", ""content"": ""<@U04JT69T00K> if we are good to use integration env then can we update in engg channel thread?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685461791.808589"", ""content"": ""<@U04JT69T00K> <@U03KLHDKL1H> - are we good on this? (I haven't checked <#C028U328HMG|engg> yet)""}, {""user"": ""askumar"", ""timestamp"": ""1685461852.056859"", ""content"": ""Yes <@U026PMDB1ND>, we upgraded Aurora cluster to latest version\nAlso I have merge aurora branch to develop as well""}]" "1685693821.105649 ","[{""user"": ""askumar"", ""timestamp"": ""1685693821.105649"", ""content"": "" <@U02BV2DGUKC> <@U026PMDB1ND>\nWhile adding @transactional to Delegate methods, some of the tests are failing.\n\nThe failures happens because the transaction is not committed in case exceptions are thrown, so the API is unable to provide response.\nThis is the same issue I suppose, which <@U026PMDB1ND> was trying to highlight in his comment\n\nI was trying to add annotation to public methods as below :\n""}, {""user"": ""aganivada"", ""timestamp"": ""1685701169.659839"", ""content"": ""<@U03KLHDKL1H> can we add a interceptor and close the transaction in failure cases?""}, {""user"": ""askumar"", ""timestamp"": ""1685702147.136769"", ""content"": ""sure <@U02BV2DGUKC>, I am trying to keep the annotation at one place.\nWill try this as well. Thanks""}]" "1684748756.014699 ","[{""user"": ""mnirmal"", ""timestamp"": ""1684748756.014699"", ""content"": ""<@U02BV2DGUKC> is it possible to get a snapshot of stage DB before the deployment? Need this to debug some PMS issue.""}, {""user"": ""aganivada"", ""timestamp"": ""1684748830.851349"", ""content"": ""sure <@U02SF36PVKL>""}, {""user"": ""aganivada"", ""timestamp"": ""1684750069.623739"", ""content"": ""taking a little longer since we have to create a DB out of snapshot to extract the data""}, {""user"": ""mnirmal"", ""timestamp"": ""1684750102.674139"", ""content"": ""Sure <@U02BV2DGUKC> no worries""}, {""user"": ""aganivada"", ""timestamp"": ""1684752567.615989"", ""content"": ""<@U02SF36PVKL> can you try the following files from ""}, {""user"": ""aganivada"", ""timestamp"": ""1684752590.596259"", ""content"": ""let me know if it works will remove the restored instance in stage""}, {""user"": ""aganivada"", ""timestamp"": ""1684752681.764999"", ""content"": "" jfyi added a db instance in stage for this request will cleanup the instance once we get a confirmation from collab""}]" "1681341591.089739 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1681341591.089739"", ""content"": ""<@U03KLHDKL1H> <@U02BV2DGUKC> - how about we write a script to generate a bunch of events of different type, users and tenants to feed into Snowflake and then try to see if we can get the queries in place for supporting the PM asks?\n\nWe can then connect the same with Retool to see if we can build the charts required. Would be neat to make some progress in this sprint, not so much on the final design but on making sure that the toolchain would give us what we need.""}, {""user"": ""aganivada"", ""timestamp"": ""1681359249.954919"", ""content"": ""<@U03KLHDKL1H> worked on building tool to dump data in Grafana on the snowflake instance, may be we can extend that to publish events?""}, {""user"": ""askumar"", ""timestamp"": ""1681359544.726969"", ""content"": ""<@U02BV2DGUKC> for snowflake I have inserted data via it's UI only where it provides option to run SQL\nI auto generated some data locally and then dumped it via snowflake dashboard \n\nWe can try doing the same for notifications as well""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681359613.795709"", ""content"": ""Sounds like a plan""}]" "1683044544.602179 ","[{""user"": ""aganivada"", ""timestamp"": ""1683044544.602179"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1683044923.252419"", ""content"": "" <@U033PPLNFRU> PFA planning sheet for 0.9.11, we are targeting to close the tasks marked in bold in sprint 3.\n\n<@U04JT69T00K> & <@U03KLHDKL1H> please update tasks un read/read-write db when you get a chance.""}, {""user"": ""askumar"", ""timestamp"": ""1683097280.026789"", ""content"": ""Updated task under Db Read/ Write:\n\n_d. Explore pg_dump approach for data replication_\n_e. CDK to populate cluster name and variables for all services_""}]" "1684755867.194579 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1684755867.194579"", ""content"": ""<@U02BV2DGUKC> - i am getting 403 forbidden for notification service in int for swagger endpoint - - is anything changed in authroizer ? cc: <@U02SF36PVKL>""}, {""user"": ""aganivada"", ""timestamp"": ""1684756136.946149"", ""content"": ""<@U0336QZAF98> did we try from VPN? ""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1684756184.097099"", ""content"": ""<@U02SF36PVKL> - can you try from VPN""}, {""user"": ""ppant"", ""timestamp"": ""1684757901.425979"", ""content"": ""Works with VPN for me""}, {""user"": ""mnirmal"", ""timestamp"": ""1684759253.302399"", ""content"": ""works. Thanks <@U0336QZAF98> <@U0431DZTPJM> <@U02BV2DGUKC>""}]" "1678387759.239149 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1678387759.239149"", ""content"": ""<@U02BV2DGUKC> - looks like release/0.9.8 has not been closed. The manifests files are not in sync with what is on develop. Is this intentional?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678388215.675579"", ""content"": "" - we are totally out of sync wrt stage.manifest.txt\n\n```Following services have mismatched versions in the environment:\n===============================================================\napp-ui - manifest version: 0.9.8-1, installed version: 0.9.8-4-hotfix-0.9.8-4\ndashboard-app-service - manifest version: 0.0.13.1, installed version: 0.0.13.3\njira-service - manifest version: 0.0.5.1, installed version: 0.0.5.2\nmessage-broker-service - manifest version: 0.0.14.1, installed version: 0.0.14.2\nonboarding-app-service - manifest version: 0.0.13, installed version: 0.0.13.1\ntoken-manager-service - manifest version: 0.0.13.1, installed version: 0.0.13\nuser-action-service - manifest version: 0.0.14.1, installed version: 0.0.14```\nLet's prevent a major incident by being careful with this.\n\n - can you please verify that the versions you are running in the env are as expected.\n\nYou can use release/0.9.8 branch as the reference for now. ""}, {""user"": ""aganivada"", ""timestamp"": ""1678421284.934929"", ""content"": ""> looks like release/0.9.8 has not been closed. The manifests files are not in sync with what is on develop. Is this intentional?\n<@U026PMDB1ND> I did close release/0.9.8 branch to main may be the git-flow plugin did not delete the branch checking""}, {""user"": ""aganivada"", ""timestamp"": ""1678421334.801939"", ""content"": ""this is where we merged release/0.9.8 with develop ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678421455.062069"", ""content"": ""May need to sync again to both develop and main and delete it on Gitlab :disappointed:""}, {""user"": ""aganivada"", ""timestamp"": ""1678421492.123519"", ""content"": ""sure will do that, looks like the issue is with my git-flow command it is not removing the remote branch for some reason.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678421515.260139"", ""content"": ""If the remote branch is protected it fails to remove""}, {""user"": ""aganivada"", ""timestamp"": ""1678421518.643849"", ""content"": ""BTW recent deployments to stage happened from main so most likely main has higher version""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678421532.436199"", ""content"": ""Oh god""}, {""user"": ""aganivada"", ""timestamp"": ""1678421539.089999"", ""content"": ""> If the remote branch is protected it fails to remove\nok that might be the issue""}, {""user"": ""aganivada"", ""timestamp"": ""1678423988.808569"", ""content"": ""<@U026PMDB1ND> looks like the issue is git-flow plugin is not deleting remote branch it is deleting only locally\n\n> anil_ganivada@Anils-MacBook-Pro ~/S/A/g/b/a/scripts (main|MERGING) [1]> git-flow release finish 0.9.8\n> Switched to branch 'develop'\n> Your branch is up to date with 'origin/develop'.\n> Merge made by the 'recursive' strategy.\n> apps/core-data-service/tests/post-deploy.default | 0\n> apps/scripts/cicd-verify.py | 5 ++++-\n> core/scripts/common-core.sh | 8 ++++++--\n> 3 files changed, 10 insertions(+), 3 deletions(-)\n> create mode 100644 apps/core-data-service/tests/post-deploy.default\n> Deleted branch release/0.9.8 (was bceddde).\n> \n> Summary of actions:\n> - Latest objects have been fetched from 'origin'\n> - Release branch has been merged into 'main'\n> - The release was tagged '0.9.8'\n> - Release branch has been back-merged into 'develop'\n> - *Release branch 'release/0.9.8' has been deleted*\nso we may have to manually execute delete remote branch\n\n> git push origin --delete release/0.9.8\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678424085.980159"", ""content"": ""<@U02BV2DGUKC> - I think that after finishing the process, if you do git status, it may show that a push is required. I think a simple git push might be enough""}, {""user"": ""aganivada"", ""timestamp"": ""1678424136.337729"", ""content"": ""ok sure Rama""}, {""user"": ""aganivada"", ""timestamp"": ""1678425528.759719"", ""content"": "" Summarising discussion I had with Rama:\nGoing forward we will create a HF CDK branch right after release branch is merged, since this seems to be creating some confusion. So basically once we have prod deployment done we will delete release branch and open a HF CDK branch immediately to be used for HF deployment. The HF branch will be merged to main when we deploy HF to prod.""}]" "1687795459.476239 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1687795459.476239"", ""content"": ""<@U04JT69T00K> are we done with the tasks we identified for GL runner? Alarms, sizing up and down during releases etc?""}, {""user"": ""pjha"", ""timestamp"": ""1687803727.375749"", ""content"": ""yes <@U026PMDB1ND>, we are done with these task also added action required based on alert in the Run Book.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687803826.739419"", ""content"": ""<@U04JT69T00K> - is there any page in Confluence that describes how to scale it up and down as well?""}, {""user"": ""pjha"", ""timestamp"": ""1687804251.212469"", ""content"": ""No <@U026PMDB1ND> there is no specific confluence page for the command, I have added the commands to the run book page, should I create a separate confluence page?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687804373.839609"", ""content"": ""Yes. please""}, {""user"": ""pjha"", ""timestamp"": ""1687804388.149529"", ""content"": ""sure will create one""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687804483.167429"", ""content"": ""To avoid dups we could create a page first and link it in the runbook""}, {""user"": ""pjha"", ""timestamp"": ""1687805508.932359"", ""content"": ""I have added this to 'How To' page\n:\n:""}]" "1675146545.464339 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1675146545.464339"", ""content"": ""Not likely that we\u2019d be able to restore the logs. We can just fix them for the future ""}, {""user"": ""akasim"", ""timestamp"": ""1675146575.750959"", ""content"": ""Sure <@U026PMDB1ND>\nThanks for the update.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1675173475.732619"", ""content"": ""<@U02HQ78V9A5> UI shows that there should be logs for those days.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1675173599.734189"", ""content"": ""But the logs are missing""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1675173605.200219"", ""content"": ""Weird state""}, {""user"": ""akasim"", ""timestamp"": ""1675173744.081559"", ""content"": ""This should not at least happen in Prod.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1675173754.547039"", ""content"": ""Prod has 30 days""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1675173758.054759"", ""content"": ""that's for sure""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1675173772.487669"", ""content"": ""We normally don't use stage for any serious debugging""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1675173828.788489"", ""content"": """"}, {""user"": ""akasim"", ""timestamp"": ""1675173847.503089"", ""content"": ""Stage has 15 days but still 4-5 days logs are missing.\nQA bugs are usually updated with stage logs so it affects the dev""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1675173853.907979"", ""content"": ""Looks like we made staging have a 7 day retention to deal with some overflows at some point""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1675173862.027749"", ""content"": ""I can fix it to be 2 weeks""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1675173903.264809"", ""content"": ""As of now stage had 7d, int 10d and prod 30d""}, {""user"": ""akasim"", ""timestamp"": ""1675173907.514679"", ""content"": ""That is the reason. Yes pls <@U026PMDB1ND> 2 weeks helps""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1675173919.695039"", ""content"": ""Fixed it now to have 14d for the first two and 30d for prod""}]" "1680835914.112049 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1680835914.112049"", ""content"": ""<@U040RCBPBEC> <@U02BV2DGUKC> <@U033PPLNFRU> - I sent a mail from Drata to invite you. Not sure what the e-mail would contain. If you can find a few minutes of time to start with that e-mail and then go through this: to install the Drata agent and complete the first round of assessment, it would be great.\n\nBased on your feedback, we can kick off a wider campaign to get everyone else onboarded.\n\nPlease edit/update the instructions in the page if they are not accurate.\n\nThanks!""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680835951.714119"", ""content"": ""Shouldn't take more than 15 minutes IMO""}, {""user"": ""svummidi"", ""timestamp"": ""1680837963.083479"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1680843267.555019"", ""content"": ""I was able to get this working <@U026PMDB1ND>, instructions were accurate. Only one question I had was should we suggest to create an Apple id with company email account instead of personal account for FileVault? In my case I was connected with personal account I signed out and created a new apple id with company email id.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680844809.536449"", ""content"": ""Good point. ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680844814.741719"", ""content"": ""We should ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680844936.608119"", ""content"": ""<@U040RCBPBEC> , <@U02BV2DGUKC> - how long did it take you approximately? I think Satya went through at least one more step beyond the basic device assessment.""}, {""user"": ""aganivada"", ""timestamp"": ""1680845003.444699"", ""content"": ""it took around 15-20 mins <@U026PMDB1ND>""}, {""user"": ""aganivada"", ""timestamp"": ""1680845413.931809"", ""content"": """"}, {""user"": ""anair"", ""timestamp"": ""1680845570.631069"", ""content"": ""all done <@U026PMDB1ND> took me about 10 minutes""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680845601.285589"", ""content"": ""Nice. Thanks <@U033PPLNFRU> ""}, {""user"": ""anair"", ""timestamp"": ""1680845607.765819"", ""content"": ""<@U026PMDB1ND> except the background check (it said my org would upload the evidence)""}]" "1677672174.740919 ","[{""user"": ""ppant"", ""timestamp"": ""1677672174.740919"", ""content"": "" For core data\u2019s `vendor-customer-mapping/all/customers/paged`, JPA query specification is running into issues with SQL if distinct = true and sort filters together in any of the following is applied with -\n\u2022 customerMapping.customerTenantId\n\u2022 customerMapping.publisher.id\n\u2022 publisherMapping.vendorTenantId\n\u2022 publisherMapping.publisher.id\nInternally with distinct = true sort filters applied, there are 3-4 queries executed, and one of the intermediate ones results in `ERROR: for SELECT DISTINCT, ORDER BY expressions must appear in select list`\n\nFor instance, if sort filter is applied in `publisherMapping.publisher.id`, then the query below throws the aforementioned error because `publisherm1_.publisher_id` is not in the select list (which is a ) because the sort spec is creating an additional join even though the entity was already fetched. This can happen in other cases where multi-depth FK relations are used with sort (open issue of JPA - )\n\n```select\n distinct vendorcust0_.id as id1_18_,\n vendorcust0_.customer_mapping_id as customer2_18_,\n vendorcust0_.customer_product_instance_id as customer3_18_,\n vendorcust0_.invitation_context_id as invitati4_18_,\n vendorcust0_.publisher_mapping_id as publishe5_18_,\n vendorcust0_.vendor_product_instance_id as vendor_p6_18_ \n from\n svc_coredata_default_axm.vendor_customer_mapping vendorcust0_ \n inner join\n svc_coredata_default_axm.publisher_mapping publisherm1_ \n on vendorcust0_.publisher_mapping_id=publisherm1_.id \n where\n publisherm1_.id=1222 \n and 1=1 \n order by\n publisherm1_.publisher_id asc limit ?```""}, {""user"": ""aganivada"", ""timestamp"": ""1677683228.167709"", ""content"": ""<@U0431DZTPJM> can we post in engg channel? not sure if this is fixed elsewhere""}, {""user"": ""aganivada"", ""timestamp"": ""1677739621.790359"", ""content"": ""<@U0431DZTPJM> did we find a solution for this issue? is this blocking apps?""}, {""user"": ""ppant"", ""timestamp"": ""1677739953.668939"", ""content"": ""No <@U02BV2DGUKC>, will repost it on engg""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1677745350.214929"", ""content"": ""<@U02BV2DGUKC> <@U0431DZTPJM> - one of the option is visibility library where we can build our own raw query which avoids this double left join :wink: This is not advertisement""}, {""user"": ""aganivada"", ""timestamp"": ""1677747894.238649"", ""content"": ""hmmm makes sense <@U0336QZAF98> , <@U0431DZTPJM> can we try the visibilyt lib route. Also worst case can we add methods with native queries?""}, {""user"": ""ppant"", ""timestamp"": ""1677748315.656309"", ""content"": ""<@U0336QZAF98> In visibility library, do we manually build JPA spec unlike in core-data or just write raw queries?""}, {""user"": ""ppant"", ""timestamp"": ""1677748867.669509"", ""content"": ""<@U02BV2DGUKC> This is not blocking apps right now because the api response for getCustomers will be distinct in itself, but asking this just for implementation perspective because if someone uses this then it throws 500""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1677748896.379769"", ""content"": ""We write raw queries..can you point me to the swagger api""}, {""user"": ""ppant"", ""timestamp"": ""1677748943.650519"", ""content"": ""This is the one - ""}, {""user"": ""ppant"", ""timestamp"": ""1677749018.096949"", ""content"": ""Here is a curl that fails in this case\n```curl -X 'GET' \\\n '' \\\n -H 'accept: application/json' \\\n -H 'Authorization: Bearer '```""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1677754819.941479"", ""content"": ""<@U0431DZTPJM> <@U02BV2DGUKC> - do you see problem in restricting sort filter - bcoz based on rsql we can have any combination of query filters - but for sort filter we can say we wil support only root table column id""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1677754864.077699"", ""content"": ""For immediate requirement""}, {""user"": ""ppant"", ""timestamp"": ""1677757024.137839"", ""content"": ""<@U0336QZAF98> Not immediate requirement right now. Sort filters are working but this error only happens if distinct=true is applied on the 4 mentioned fields. For this API specifically distinct is not required but the problem can come in some other new use case because apps usually send distinct=true by default in their calls""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1677757172.549849"", ""content"": ""bcoz distinct=true is default value - restricting what sortfilter user can pass can be controlled for sometime - can you check with apps - on various sort clauses they are planning to use""}, {""user"": ""ppant"", ""timestamp"": ""1677757214.120689"", ""content"": ""Sure, will check""}]" "1691647134.784389 ","[{""user"": ""aganivada"", ""timestamp"": ""1691647134.784389"", ""content"": ""<@U02D4DUKDQC> <@U02HQ78V9A5> deployed MB with whitelisting can we check now?""}, {""user"": ""akasim"", ""timestamp"": ""1691647556.658199"", ""content"": ""Subscription renewal started again in MB <@U02BV2DGUKC>\ncc: <@U02D4DUKDQC>""}, {""user"": ""aganivada"", ""timestamp"": ""1691652489.963669"", ""content"": ""<@U02HQ78V9A5> deployed the MB change can we check now?""}, {""user"": ""aganivada"", ""timestamp"": ""1691652496.790589"", ""content"": ""cc: <@U02D4DUKDQC>""}, {""user"": ""akasim"", ""timestamp"": ""1691652625.512319"", ""content"": ""Brokering works now <@U02BV2DGUKC>\ncc: <@U03NZ7Z52S2> <@U02D4DUKDQC>""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1691652701.588119"", ""content"": ""<@U02BV2DGUKC> let us update the release thread with these findings""}, {""user"": ""aganivada"", ""timestamp"": ""1691652728.126549"", ""content"": ""sure <@U02D4DUKDQC>""}, {""user"": ""aganivada"", ""timestamp"": ""1691664557.318839"", ""content"": ""<@U04JT69T00K> here is the filter I used for all Event bridge invocation failures ""}]" "1684993369.283339 ","[{""user"": ""vtiwari"", ""timestamp"": ""1684993369.283339"", ""content"": "" For accessing tenany and other api, it says This workspace has no available licenses. was logging in with microsoft account.""}, {""user"": ""ppant"", ""timestamp"": ""1684993412.641729"", ""content"": ""Reopen the browser and try again. It happened before also""}, {""user"": ""aganivada"", ""timestamp"": ""1684993562.830009"", ""content"": ""<@U04RUV6SGU9> did it work on refresh?""}, {""user"": ""vtiwari"", ""timestamp"": ""1684993650.584869"", ""content"": ""<@U02BV2DGUKC> no, it did not.""}, {""user"": ""aganivada"", ""timestamp"": ""1684993698.225929"", ""content"": ""can you try now?""}, {""user"": ""aganivada"", ""timestamp"": ""1684993706.298289"", ""content"": ""just added more licenses""}, {""user"": ""aganivada"", ""timestamp"": ""1684993741.838139"", ""content"": ""> Reopen the browser and try again. It happened before also\n<@U0431DZTPJM> if only getting additional licenses was this easy :sweat_smile:""}, {""user"": ""ppant"", ""timestamp"": ""1684993794.786499"", ""content"": ""It happened with Mohith a couple of days back. He tried again and it worked. I thought it might be a caching issue or something :sweat_smile: Life isn\u2019t easy""}, {""user"": ""aganivada"", ""timestamp"": ""1684993878.923689"", ""content"": ""ok may be I added some license just when Mohit was trying :slightly_smiling_face:""}, {""user"": ""vtiwari"", ""timestamp"": ""1684994441.923049"", ""content"": ""it worked <@U02BV2DGUKC>, thanks.""}]" "1684228947.276629 ","[{""user"": ""ppant"", ""timestamp"": ""1684228947.276629"", ""content"": "" <@U0336QZAF98> The pipeline for apps-workflow-lambda is failing at create-pypi stage because a wheel with the same package name already exists Shall we go ahead and delete the already existing wheel? We do the same thing for plat-libs-epoch also""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1684229115.608439"", ""content"": ""Actually you can trigger deletion from pipeline - rather than doing manually everytime - refer ""}, {""user"": ""ppant"", ""timestamp"": ""1684229203.718359"", ""content"": ""Yeah <@U0336QZAF98>, we can add this in the pipeline. Also since its a lambda, unlike plat-libs-epoch, its package is not imported anywhere. Will it be a better option to remove this stage from the pipeline altogether?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1684229228.999969"", ""content"": ""Typically - if it is lambda we dont publish to pypi""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1684229253.247329"", ""content"": ""do u check - only if it is passed then only publish\n```if [ \""$PUBLISH_PYPI\"" == \""1\"" ]```""}, {""user"": ""ppant"", ""timestamp"": ""1684229295.798679"", ""content"": ""This variable comes == 1 if the branch is protected so this stage always gets executed in develop branches""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1684229310.149199"", ""content"": ""No\u2026for lambda we dont set""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1684229349.126909"", ""content"": """"}, {""user"": ""ppant"", ""timestamp"": ""1684229355.930769"", ""content"": ""Ok, got it. This is set == 1 for apps lambda""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1684229358.197789"", ""content"": ""you can this is set explicily to zero""}, {""user"": ""ppant"", ""timestamp"": ""1684229422.983459"", ""content"": ""Thanks, will do :+1:""}]" "1687539640.871109 ","[{""user"": ""pjha"", ""timestamp"": ""1687539640.871109"", ""content"": ""<@U026PMDB1ND> We have created a policy to restrict cli and console access to the *IAM user* if the user is not connected to the VPN, wanted the check if we can add the policy to the in INT ? c.c <@U02BV2DGUKC>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687795264.109109"", ""content"": ""<@U04JT69T00K> <@U02BV2DGUKC> - we can do this but let\u2019s experiment with this during IST hours that don\u2019t impact a lit of people and test that it is working well. Also document it well and provide instructions for rollback in the worst case.\n\nJust check that GitLab tasks don\u2019t fail after this change ""}, {""user"": ""aganivada"", ""timestamp"": ""1687795414.081809"", ""content"": ""Sure thank you <@U026PMDB1ND>, we missed considering gitlab will test during low impact ist hours.""}]" "1678193171.898769 ","[{""user"": ""aganivada"", ""timestamp"": ""1678193171.898769"", ""content"": "" can someone connect to VPN and check if you can access ?\n\n\nprovisioned a route53 resolver in int and updated private DNS settings in perimeter 81. Now once we are on VPN we should be able to access services directly with private domain name we don't have to make any additional changes to dns settings in mac.""}, {""user"": ""ppant"", ""timestamp"": ""1678193522.520259"", ""content"": ""I am able to access. Is it an http endpoint right now?""}, {""user"": ""aganivada"", ""timestamp"": ""1678193585.293349"", ""content"": ""thank you <@U0431DZTPJM>, it is http since we are accessing from within vpn this is similar to how other services interact with tenancy""}, {""user"": ""svummidi"", ""timestamp"": ""1678766575.487879"", ""content"": ""<@U02BV2DGUKC> - <@U02Q02G09QX> is not able to access tenancy service, is it anything related to this thread?\nEven for me also service is not accessible\n""}, {""user"": ""aganivada"", ""timestamp"": ""1678766677.190839"", ""content"": ""sorry <@U040RCBPBEC> I am working on the vpn thing""}, {""user"": ""aganivada"", ""timestamp"": ""1678766681.889959"", ""content"": ""let me update""}, {""user"": ""aganivada"", ""timestamp"": ""1678766744.383459"", ""content"": ""should be accessible now <@U040RCBPBEC>""}, {""user"": ""aganivada"", ""timestamp"": ""1678767357.149839"", ""content"": ""<@U02Q02G09QX> shared details with you 1-1""}]" "1677582111.906139 ","[{""user"": ""aganivada"", ""timestamp"": ""1677582111.906139"", ""content"": ""<@U03RSS0S76Y> <@U02D4DUKDQC> as part of automating publishing of bot to s3 bucket are we also planning to push images/bot icons from gitlab to s3?""}, {""user"": ""snangia"", ""timestamp"": ""1677583091.064379"", ""content"": ""Yes, it will be a zipped file containing the icon.""}, {""user"": ""aganivada"", ""timestamp"": ""1677583574.290399"", ""content"": ""ok is it applicable only for teams images or slack also ?""}, {""user"": ""snangia"", ""timestamp"": ""1677583614.144969"", ""content"": ""for slack, yet to check if we'll be using s3""}]" "1680770423.709189 ","[{""user"": ""askumar"", ""timestamp"": ""1680770423.709189"", ""content"": ""\nReference doc for Aurora integration and testing.\nMore details will be added as we proceed, please feel free to add your comments/questions\n""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1680791276.394229"", ""content"": ""<@U03KLHDKL1H> I have put my questions in below link - it will be great if you can compare with this library and see what are pros and cons - if we go with oour own approach vs library\n""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1680791345.395059"", ""content"": "" - on how to use the library""}]" "1682316318.065599 ","[{""user"": ""akalyta"", ""timestamp"": ""1682316318.065599"", ""content"": ""<@U02BV2DGUKC> could you please raise the hard limit for OpenAI? :) We're at $139 now but we have a week left and I want to avoid having interruptions with OpenAI. I think $180 would be good, thank you!!""}, {""user"": ""aganivada"", ""timestamp"": ""1682317464.936959"", ""content"": ""sure <@U02Q02G09QX>""}, {""user"": ""akalyta"", ""timestamp"": ""1682317483.114469"", ""content"": ""thank you very much <@U02BV2DGUKC>!""}, {""user"": ""aganivada"", ""timestamp"": ""1682317663.490469"", ""content"": ""<@U0336QZAF98> /<@U02Q02G09QX> updated hard limit to $180 we have also raised a request to raise account limit to $ 250 so we can go even higher if it makes sense.""}]" "1687173596.532659 ","[{""user"": ""aganivada"", ""timestamp"": ""1687173596.532659"", ""content"": ""<@U026PMDB1ND> logz followed up on the field mapping issue today. They are mentioning that text(analyzed) field which is required to understand patterns from logs doesn't work well with regex. They have suggested some changes to regex but they are saying that the regex results on text(analyzed) fields are not reliable until bug is fixed. So we have 2 options\n\n1. Revert the field type to keyword(string) in all 3 env's if we want to use regex (we will not have logz ability to understand patterns from logs), Wait for logz to reachout once bug is fixed and then change field to type text(analyzed)\n2. use text(analyzed) but change the regex expression to include lower and upper case e.g: /.*[Gg][Ee][Tt].*/ \nme=>\n```Hi Or/Naama, thank you for your response. We updated this field to Text(Analyzed) based on recommendation from logz since we wanted Pattern analysis on this field. Not sure when the field got updated back to keyword(string) in int. Is there any approach to have the regex as well as log-patterns working on message_obj.msg field? ```\nlogz =>\n```A tokenized field means that data is stored under a token, and in OpenSearch, in some cases, two separated words are kept under the same token since spaces and other special keys are not stored in OpenSearch as well, which means that for example, for the String \""not found\"" could be stored as well as \""notfound\"" and other combinations. And Regex (/.*not found.*/) won't work as expected because it's checking if there is a match for the regex phrase per token, not per the entire string value.\n\nI went over this chat, and I do see that for account 389410 (Staging environment), the field \""message_obj.msg\"" is mapped as a Text (Analysed Field):\n\nAnd for account 389405 (Integration environment) which was stated that it is working it is mapped as a String field:\n\nYou can use regex with \""message_obj.msg\"" when it is a Text (Analysed Field) type field, but the results may not be accurate based on my explanation above.\nAlso, we also have an open bug currently that regex on analyzed fields works only with lower cased characters due to the analyzer saving the tokens lower cased we have linked this chat to the open bug ticket so once and if it is resolved, we will reach out.\nBut, you can use the following example on how to use regex with the field \""message_obj.msg\"" in account \"" 389410 (Staging environment)\"" when providing the regex to include upper case and lower case options:\nmessage_obj.msg:/.*[Gg][Ee][Tt].*/ \n```\ncc: <@U02D4DUKDQC> ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687190491.573079"", ""content"": ""<@U02BV2DGUKC> I think we a bunch of alerts (& bypasses) defined on the regex pattern. So, it is important for us to have regex working. I think that it is a pain to go upper/lower for every character.\n\nWe should ask if we can move to keyword for now and then after the bug is fixed move back to text since we don't rely on the pattern analysis as much (right <@U02D4DUKDQC>?)\n\nWe should also ask them for the bug id so that we can check down the line.""}, {""user"": ""aganivada"", ""timestamp"": ""1687193018.022889"", ""content"": ""sure <@U026PMDB1ND> updating to keyword is straightforward I can update it tmrw morning IST (to avoid any risk with indexing). Will check on the bug-id with them.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687201180.929139"", ""content"": ""Anil, we should see if there would be any problem in going back to text before updating to KW""}, {""user"": ""aganivada"", ""timestamp"": ""1687349992.904099"", ""content"": ""<@U026PMDB1ND> got an update from logz they mentioned there should not be any issue swapping the field. However, they proposed if we can use \"".raw\"" extension to use regex. I thought this may not work for us since we'd have to not only update all existing alerts and dashboard but also remember to use this for all future queries. So for now updating field to keyword in stage and prod.\n\nthey also shared the bug-id 61838 for the regex issue.""}, {""user"": ""aganivada"", ""timestamp"": ""1687350084.784689"", ""content"": ""updated in stage will wait for some time to make sure there are no parsing issues and update prod""}, {""user"": ""aganivada"", ""timestamp"": ""1687354667.799119"", ""content"": ""updated prod mapping also and regex seems to be working""}]" "1683045441.298179 ","[{""user"": ""pjha"", ""timestamp"": ""1683045441.298179"", ""content"": "" <@U026PMDB1ND> I have been working on creating RDS Aurora Serverless_v2 instance using cdk_v2, I am facing an issue where cluster is created but could create read instances . There are ways to attach non-serverless instances, but could not find ways to attach serverless instances.\n Please refer does't look like it's fully supported.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683090359.628469"", ""content"": ""<@U04JT69T00K> - are you saying that we have an issue creating a serverless v2 cluster using CDK2?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683090367.429929"", ""content"": ""Just trying to understand what is broken""}, {""user"": ""pjha"", ""timestamp"": ""1683090500.153719"", ""content"": ""yes, I couldn't create cluster with the read and write instances using cdk""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683091946.340579"", ""content"": """"}]" "1687150990.549739 ","[{""user"": ""mnirmal"", ""timestamp"": ""1687150990.549739"", ""content"": "" can you please deploy - core-stack sns_events_publisher_stack.py collab-pulse-notification-events-sqs from release/0.9.12. I'm getting this error -\n```Traceback (most recent call last):\n File \""/Users/mohith01/cdk-artifacts/base-infra/core/cdk_app.py\"", line 760, in <module>\n main_app()\n File \""/Users/mohith01/cdk-artifacts/base-infra/core/cdk_app.py\"", line 739, in main_app\n provision_es_initial_setup_stack(app, envw, base_stacks)\n File \""/Users/mohith01/cdk-artifacts/base-infra/core/cdk_app.py\"", line 420, in provision_es_initial_setup_stack\n ESInitialSetupStack(\n File \""/Users/mohith01/cdk-artifacts/base-infra/venvs/.venv.52440400/lib/python3.9/site-packages/jsii/_runtime.py\"", line 112, in __call__\n inst = super().__call__(*args, **kwargs)\n File \""/Users/mohith01/cdk-artifacts/base-infra/core/stacks/es_initial_setup_stack.py\"", line 44, in __init__```""}, {""user"": ""aganivada"", ""timestamp"": ""1687152040.968779"", ""content"": ""ack <@U02SF36PVKL>""}, {""user"": ""aganivada"", ""timestamp"": ""1687152071.602519"", ""content"": ""<@U02SF36PVKL> are you trying to deploy in int?""}, {""user"": ""mnirmal"", ""timestamp"": ""1687152097.675969"", ""content"": ""yes <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1687152117.527659"", ""content"": ""if yes then please browse to ../es-cluster-initial-setup/ and run ./build-package.sh""}, {""user"": ""aganivada"", ""timestamp"": ""1687152136.473139"", ""content"": ""same thing to be done for ../ip_authorizer/""}, {""user"": ""aganivada"", ""timestamp"": ""1687152144.069749"", ""content"": ""and then you can run the deloy command""}, {""user"": ""mnirmal"", ""timestamp"": ""1687153676.285429"", ""content"": ""<@U02BV2DGUKC> can you deploy this on stage?""}, {""user"": ""aganivada"", ""timestamp"": ""1687153686.315149"", ""content"": ""sure <@U02SF36PVKL>""}, {""user"": ""aganivada"", ""timestamp"": ""1687153701.866669"", ""content"": ""did we push changes to release branch?""}, {""user"": ""mnirmal"", ""timestamp"": ""1687153721.353519"", ""content"": ""yes <@U02BV2DGUKC>""}, {""user"": ""mnirmal"", ""timestamp"": ""1687153725.203019"", ""content"": ""pushed it""}, {""user"": ""aganivada"", ""timestamp"": ""1687153752.068239"", ""content"": ""this one?""}, {""user"": ""aganivada"", ""timestamp"": ""1687153754.566459"", ""content"": """"}, {""user"": ""mnirmal"", ""timestamp"": ""1687153764.547609"", ""content"": ""yes <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1687153773.376519"", ""content"": ""deploying now""}, {""user"": ""aganivada"", ""timestamp"": ""1687154704.495569"", ""content"": ""done <@U02SF36PVKL>\n\n""}, {""user"": ""mnirmal"", ""timestamp"": ""1687154725.125249"", ""content"": ""Thank you <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1687155883.241039"", ""content"": ""<@U02SF36PVKL> can we update deployment dependency page on this? ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687188778.216149"", ""content"": ""<@U02SF36PVKL> - is the update to the page done?""}, {""user"": ""mnirmal"", ""timestamp"": ""1687192861.157699"", ""content"": ""Sorry forgot to update <@U02BV2DGUKC> and <@U026PMDB1ND> - \""sns-events-publisher\"" is already part of the core-stacks deployment dependencies. So need for any extra steps.""}, {""user"": ""aganivada"", ""timestamp"": ""1687193875.704049"", ""content"": ""<@U02SF36PVKL> don't we need collab-pulse-notification-events-sqs ?""}, {""user"": ""mnirmal"", ""timestamp"": ""1687193985.078669"", ""content"": ""we don't <@U02BV2DGUKC> - <@U03BPNY5AGM> can you confirm if we need to tdeploy the sqs also for adding a new event?""}]" "1683749391.715059 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1683749391.715059"", ""content"": ""<@U02BV2DGUKC> <@U04JT69T00K> - were we planning on allowing folks to connect to RDS instances in an env after connecting to the VPN? Right now, it is not working.""}, {""user"": ""aganivada"", ""timestamp"": ""1683775770.021509"", ""content"": ""ack <@U026PMDB1ND> , looks like we need to add P81 SG to rds access will add this as part of VPN CDK changes.""}]" "1692806479.254729 ","[{""user"": ""pjha"", ""timestamp"": ""1692806479.254729"", ""content"": "" <@U026PMDB1ND> please review weekly report analysis using CUR for stage c.c <@U02BV2DGUKC>""}, {""user"": ""pjha"", ""timestamp"": ""1692806657.517449"", ""content"": ""when the cost is less than $2 in a week the graph excludes the bar.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692807770.768009"", ""content"": ""<@U04JT69T00K> - is there the raw data somewhere?""}, {""user"": ""pjha"", ""timestamp"": ""1692808154.278689"", ""content"": ""<@U026PMDB1ND> yes, CUR uploads raw data to s3.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692810281.614969"", ""content"": ""<@U04JT69T00K> - can you give me aws cli cmd to get the s3 data?""}, {""user"": ""pjha"", ""timestamp"": ""1692856936.075879"", ""content"": ""<@U026PMDB1ND> use below command to get the s3 data\naws s3 cp . --profile axmstage\n\nHere I am not directly querying this file instead querying Athena to get response.\n\nCUR provides the cloudformation template which creates Lambda Function, Athena and AWS Glue Database. For any new event to the s3 bucket lambda function is triggered which then update the GLUE.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692895862.134059"", ""content"": ""<@U04JT69T00K> - as part of completing this task, please put together a Confluence page with the details of the flow, some sample queries and such. Thank you!""}, {""user"": ""pjha"", ""timestamp"": ""1692939758.849999"", ""content"": ""sure <@U026PMDB1ND>""}]" "1681325132.393469 ","[{""user"": ""akalyta"", ""timestamp"": ""1681325132.393469"", ""content"": ""<@U0281D3GWHL> [G2 data parsing]\nHi! Could you please describe how you are parsing the G2 products data? I am curious if you could make some changes to the parsed files to make sure they interact well with all the scripts that use this data :)\n\nThe changes I had in mind:\n1. Uploading files in .csv format would be best\n2. Please upload the files directly to `rawdata` without creating a new folder\n3. Please ensure that filenames are all lowercase, if product name consists of 2+ words please use dash \""-\"" to separate the words\n4. (Update: please ignore this item) In the data file itself, could you please name the columns as the fields in G2? (\""What do you like about the product?\"" vs \""PROS\"") \n<@U0281D3GWHL> you mentioned that you're avoiding CSV format to ensure that commas in the reviews don't cause formatting issues and that you're also parsing the data from Capterra, right?\nDepending on how you're parsing you might actually be ok with commas in responses - if you're using `beautifulsoup` from Python it should work well with commas, the same thing if you're populating .csv file manually using GSheets, MSExcel, or Numbers :) Are you also collecting the reviews from G2 and Capterra for the same products and putting them into the same excel file?""}, {""user"": ""svummidi"", ""timestamp"": ""1681340754.860359"", ""content"": ""<@U0281D3GWHL> Even if we are using Excel for collaboration between different users, before uploading to S3 if we can convert to CSV, I think that is sufficient.\n<@U02Q02G09QX> - We are planning to collect data from different sources, so we don\u2019t want to make it very specific to one site.\n> 1. In the data file itself, could you please name the columns as the fields in G2? (\u201cWhat do you like about the product?\u201d vs \u201cPROS\u201d)\n""}, {""user"": ""akalyta"", ""timestamp"": ""1681340787.863109"", ""content"": ""Thank you <@U040RCBPBEC>, this is clear now :)""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1681357509.987119"", ""content"": ""Alright, so the ask is just to upload a CSV rather than a xlsx and we upload it to the 'rawdata' folder without any changes to the format of the csv?""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1681454043.784949"", ""content"": ""<@U040RCBPBEC> <@U02Q02G09QX> I have uploaded two CSVs to rawdata/, one for Splunk Enterprise and the other for all other Splunk Products. Let me know if this works or you need any other changes""}, {""user"": ""akalyta"", ""timestamp"": ""1681496705.054699"", ""content"": ""Thank you <@U0281D3GWHL>! This works great, just make sure that file name in all lower case and dash-separated for the next files :) Great data!""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1681565840.653609"", ""content"": ""Will do, thanks""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1681976806.879839"", ""content"": ""<@U02Q02G09QX> <@U0336QZAF98> uploaded zscaler.csv, FYI.""}, {""user"": ""svummidi"", ""timestamp"": ""1682350876.798569"", ""content"": ""<@U02Q02G09QX> <@U0336QZAF98> - Are we able to create products and link this data using our chat interface so we can explore this data?""}, {""user"": ""akalyta"", ""timestamp"": ""1682354680.818599"", ""content"": ""So far I was manually populating indexes due to low data volume. I can add a quick automation script if necessary today :)""}, {""user"": ""svummidi"", ""timestamp"": ""1682354938.732239"", ""content"": ""<@U02Q02G09QX> I don\u2019t think we have full automation to create product instance ID in tenant. but some script with tenant ID and product instance ID, helps instead of downloading manually generating and uploading.\nPlease make sure there is a product instance for each product in tenant 44077 and make it available for others.""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1682355165.005519"", ""content"": ""I have uploaded crowdstrike.csv and wiz.csv to S3 now""}, {""user"": ""akalyta"", ""timestamp"": ""1682356591.951749"", ""content"": ""will do <@U040RCBPBEC> :)""}]" "1676444971.982239 ","[{""user"": ""aganivada"", ""timestamp"": ""1676444971.982239"", ""content"": ""<@U0431DZTPJM> got another status page tracking solution that would work at scale from <@U026PMDB1ND> can we evaluate a few options and check the one that would suite us""}, {""user"": ""ppant"", ""timestamp"": ""1676445265.387289"", ""content"": ""Sure anil. I will start tracking them in a confluence page""}, {""user"": ""aganivada"", ""timestamp"": ""1676445965.834519"", ""content"": ""<@U0431DZTPJM> let's hold on researching on status page I sent a message to Aryan on more specific requirements, we can proceed once we get confirmation that this is what we are looking for.""}]" "1684217043.677469 ","[{""user"": ""aganivada"", ""timestamp"": ""1684217043.677469"", ""content"": ""<@U03KLHDKL1H> can you review specifically changes related to aurora that we merged to develop branch? I pulled latest changes from develop to cdk v2 and found some conflicts WRT aurora""}, {""user"": ""askumar"", ""timestamp"": ""1684218312.217239"", ""content"": ""Thanks <@U02BV2DGUKC>..All changes are there""}, {""user"": ""aganivada"", ""timestamp"": ""1684222162.976229"", ""content"": ""<@U04JT69T00K> please let me know one we update the 3 stacks we can merge cdkv2 to develop""}, {""user"": ""pjha"", ""timestamp"": ""1684222181.205219"", ""content"": ""yes, sure working on the last one""}, {""user"": ""pjha"", ""timestamp"": ""1684223286.971939"", ""content"": ""<@U02BV2DGUKC> merged my changes to feature/PLAT-792""}, {""user"": ""aganivada"", ""timestamp"": ""1684226107.209189"", ""content"": ""thank you <@U04JT69T00K>""}]" "1690589930.189259 ","[{""user"": ""svummidi"", ""timestamp"": ""1690589930.189259"", ""content"": "" <@U02BV2DGUKC> <@U03KLHDKL1H> - Can you help with below error?\n<@U02Q02G09QX> Running a POC service in one of the EC2 instance trying to access secrets manager. It used to work fine before, now it stopped working with below error.\n```023-07-29 00:08:46,667 DEBUG:Event needs-retry.secrets-manager.GetSecretValue: calling handler <botocore.retryhandler.RetryHandler object at 0x7f45ff064a60>\n2023-07-29 00:08:46,667 DEBUG:No retry needed.\nTraceback (most recent call last):\n File \""/home/ec2-user/llama-index-demo/pulseDemo/app_api.py\"", line 21, in <module>\n RESPONSE = secret_client.get_secret_value(SecretId=AppConfig.OPENAI_SECRET_ARN)\n File \""/home/ec2-user/llama-index-demo/pulseDemo/llama-index-env/lib64/python3.9/site-packages/botocore/client.py\"", line 530, in _api_call\n return self._make_api_call(operation_name, kwargs)\n File \""/home/ec2-user/llama-index-demo/pulseDemo/llama-index-env/lib64/python3.9/site-packages/botocore/client.py\"", line 960, in _make_api_call\n raise error_class(parsed_response, operation_name)\nbotocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:iam::433798924509:user/akalyta is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:us-west-2:433798924509:secret:llama-index-demo-openai-secret-rMswO9 with an explicit deny in an identity-based policy```""}, {""user"": ""svummidi"", ""timestamp"": ""1690590111.258859"", ""content"": ""We are accessing this from EC2 instance. We need this as a backup option until the current release completed.\n""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690605297.417539"", ""content"": "" - I think \u201carn:aws:iam::433798924509:user/akalyta\u201d - this is bcoz of vpn access - we added explicit vpn access for user based identity i guess""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690605316.022219"", ""content"": ""May be we have to switch to role based access here""}, {""user"": ""aganivada"", ""timestamp"": ""1690606188.873649"", ""content"": ""checking <@U040RCBPBEC>\n\n<@U0336QZAF98> policies are always on roles I don't think we have user specific policies all VPN policies are role based. if we are running on ec2 instance why is it taking user name shouldn't the arn be ec2-user or resource id?""}, {""user"": ""aganivada"", ""timestamp"": ""1690606390.635389"", ""content"": ""~<@U0336QZAF98> /<@U040RCBPBEC> can you point me to the ec2 instance? is it the ExampleAppServerInstance ?~""}, {""user"": ""aganivada"", ""timestamp"": ""1690606407.773389"", ""content"": ""please ignore""}, {""user"": ""aganivada"", ""timestamp"": ""1690606728.704919"", ""content"": ""<@U040RCBPBEC> this instance seems to be in n.virginia and not in the same vpc & region so the vpn rule is getting enforced since an external request is being made and we have a policy attached to axm users group (Artem is part of it) where we enforce access only through vpn.\n\n```{\n \""Version\"": \""2012-10-17\"",\n \""Statement\"": [\n {\n \""Sid\"": \""VisualEditor0\"",\n \""Effect\"": \""Deny\"",\n \""Action\"": \""*\"",\n \""Resource\"": \""*\"",\n \""Condition\"": {\n \""NotIpAddress\"": {\n \""aws:SourceIp\"": [\n \""157.245.106.14/32\"",\n \""131.226.33.86/32\""\n ]\n },\n \""Bool\"": {\n \""aws:ViaAWSService\"": \""false\""\n }\n }\n }\n ]```""}, {""user"": ""aganivada"", ""timestamp"": ""1690606782.370259"", ""content"": ""let me check if we can do something to bypass this case""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690606825.402199"", ""content"": ""Why would we want to run anything outside the vpc?""}, {""user"": ""aganivada"", ""timestamp"": ""1690606851.758769"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1690606918.291719"", ""content"": ""yeah ideally if the instance is in vpc and if we use the resource arn to read secret instead of user then it should work""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690606929.272819"", ""content"": ""How hard would it be to run this in our vpc?""}, {""user"": ""aganivada"", ""timestamp"": ""1690607276.211219"", ""content"": ""+1, <@U040RCBPBEC> / <@U0336QZAF98> / <@U02Q02G09QX> can we provision a new instance and move the code to our integration vpc?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690677195.322279"", ""content"": "" LMK if you need some help to save some cycles. I think moving it to us-west-2 reduces future hassles. Next time when we spin something up, it might be better to just be deliberate about it or ask the platform team to set it up if we can wait a little bit.""}, {""user"": ""svummidi"", ""timestamp"": ""1690846452.226489"", ""content"": ""<@U026PMDB1ND> <@U02BV2DGUKC> - First, I considered abandoning this EC2 instance altogether. On second thought, it is better to keep it as a backup option, at least in INT.\nCan you start a new EC2 instance? Probably we need it only for one or two weeks. This instance needs to access S3 and the secrets manager. We need the ability to access this instance from INT.\nWe are taking backup from the old instance. We will start the service on this new instance, once it is ready.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690853211.450609"", ""content"": ""<@U04JT69T00K> - can you please help with this? This time we'd do it manually, but we should also set up a CDK stack that can be allows the type of the instance to be configured and when run would create an instance in the correct SG with the required key pairs etc.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1691090236.678239"", ""content"": ""<@U02Q02G09QX> - do you know if this has been taken care of?""}, {""user"": ""svummidi"", ""timestamp"": ""1691090570.717209"", ""content"": ""<@U026PMDB1ND> I don\u2019t know any new update - <@U04JT69T00K> did you get a chance to start the instance?""}, {""user"": ""pjha"", ""timestamp"": ""1691122550.829069"", ""content"": ""<@U040RCBPBEC> Not yet, I will work on it""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1691129620.728579"", ""content"": ""<@U04JT69T00K> it would be nice if we can prioritize this""}, {""user"": ""pjha"", ""timestamp"": ""1691129688.930099"", ""content"": ""<@U026PMDB1ND> sure I will take it on priority""}, {""user"": ""pjha"", ""timestamp"": ""1691131888.639189"", ""content"": ""<@U040RCBPBEC> <@U026PMDB1ND> I have created EC2 instance in with s3 and Secret access. Please provide your ssh public key.""}, {""user"": ""aganivada"", ""timestamp"": ""1691132995.103659"", ""content"": ""cc: <@U02Q02G09QX>\n\n<@U04JT69T00K> can you also share the confluence page with instructions on generating public key?""}, {""user"": ""pjha"", ""timestamp"": ""1691133291.479569"", ""content"": ""*command to generate ssh key :-* ssh-keygen -t ed25519 -C <email_id> -f <file_path>\n<@U02BV2DGUKC> here this is not for temporary access so above command will do, we don't need specific file name for the keys.""}, {""user"": ""pjha"", ""timestamp"": ""1691133473.299259"", ""content"": ""We can also follow script to generate the key :""}]" "1682308685.484719 ","[{""user"": ""aganivada"", ""timestamp"": ""1682308685.484719"", ""content"": ""<@U03NZ7Z52S2> during regression cycle can we run the tests we identified for permissions validation during 0.9.9-hf this is required since we did some refactoring and added some changes to RBAC to enable reporting on vendor side. cc: <@U03DHUAJVMK>""}, {""user"": ""bganganna"", ""timestamp"": ""1682309776.066769"", ""content"": ""<@U02BV2DGUKC> Do we need to test from customer side as well or vendor side onboarding is enough?""}, {""user"": ""aganivada"", ""timestamp"": ""1682310076.959919"", ""content"": ""vendor side will cover both flows but if we have time can we cover both scenarios? it will give us more confidence. Also collaborators flow is more important in context of the RBAC change on either side""}]" "1673888228.254499 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1673888228.254499"", ""content"": ""<@U02BV2DGUKC> If we add an api in coredata to match this requirement would it reduce the round trips?""}, {""user"": ""aganivada"", ""timestamp"": ""1673888554.616309"", ""content"": ""Rama, we will need info about the organisation from tenancy so we can't move the call to core but we can update tenancy call to return open invitations to reduce the two calls to one paginated api. \n\nAlso I think it will be better if we split these into 2 components in UI open invitations and active customers. Will have to check with Aryan on that.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673908461.148199"", ""content"": ""Yeah.. I have seen both kinds of presentation where all the entities are in shown with a status column (typically users), but also came across tabbed UI for current and invited (not yet online).\n\nIf we are not supporting any sort of filtering on the top of the page tabbed view with two groups (invited and connected customers) might be better""}]" "1679321897.804749 ","[{""user"": ""aganivada"", ""timestamp"": ""1679321897.804749"", ""content"": ""<@U04JT69T00K> any luck with load-balancer research?""}, {""user"": ""pjha"", ""timestamp"": ""1679322188.802149"", ""content"": ""not yet, was stuck with the other issues related to bringing up multiple instance, since those are fixed now I will start with the looking into the load-balancer thing.""}]" "1692878563.301209 ","[{""user"": ""aganivada"", ""timestamp"": ""1692878563.301209"", ""content"": ""<@U04JT69T00K> can we check why we crossed upto 100% of CPU in GL-runner instance. should we reduce number of parallel pipelines running? cc: <@U02HQ78V9A5>""}, {""user"": ""pjha"", ""timestamp"": ""1692878605.179899"", ""content"": ""<@U02BV2DGUKC> yes I am looking into this, also there is an alert for disk space""}, {""user"": ""aganivada"", ""timestamp"": ""1692878797.694809"", ""content"": ""ok""}, {""user"": ""pjha"", ""timestamp"": ""1692879368.692399"", ""content"": ""<@U02BV2DGUKC> I can see in CloudWatch total 7 jobs was running at the time when CPU crossed the threshold.""}, {""user"": ""aganivada"", ""timestamp"": ""1692879401.287159"", ""content"": ""hmmm would it help if we reduced the count of paralllel jobs to 5?""}, {""user"": ""aganivada"", ""timestamp"": ""1692879422.016949"", ""content"": ""100% CPU will anyways create issues for running jobs""}, {""user"": ""pjha"", ""timestamp"": ""1692879433.442019"", ""content"": ""yes, I will reduce it to 5""}]" "1690864515.078989 ","[{""user"": ""svummidi"", ""timestamp"": ""1690864515.078989"", ""content"": ""<@U0336QZAF98> - Even after bouncing the service, askai celery not picking up the latest builds. The checksum from build and checksum in ECS not matching - Can you check if any of your changes to use custom ECR causing this issue?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690865742.798009"", ""content"": ""<@U040RCBPBEC> - latest develop branch build is deployed""}, {""user"": ""svummidi"", ""timestamp"": ""1690869315.536239"", ""content"": ""<@U0336QZAF98> - Now it is showing versions as matched but it is executing old code for the task execution. Is it possible that some other cluster connected to same task queue? Let me know if you have few minutes we can discuss in a huddle.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690869982.259749"", ""content"": ""sure <@U040RCBPBEC>""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690869987.791279"", ""content"": ""QC""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690870300.563049"", ""content"": ""<@U040RCBPBEC> - if you can let me know what is missing - it will be helpful""}, {""user"": ""svummidi"", ""timestamp"": ""1690870591.627729"", ""content"": ""If i directly submit the request it is running fine.\nIf I submit the request via askAI service, it is failing. As per the traceback in celery_taskmeta, it is using old code and IP for worker is different.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690870644.445589"", ""content"": ""This is for chat api - right ?""}, {""user"": ""svummidi"", ""timestamp"": ""1690870651.160799"", ""content"": ""Yes""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690870655.122689"", ""content"": ""checking""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690870742.756509"", ""content"": ""```curl -X 'POST' \\\n '' \\\n -H 'accept: application/json' \\\n -H 'Authorization: Bearer gg' \\\n -H 'Content-Type: application/json' \\\n -d '{\n \""node\"": {\n \""id\"": 0,\n \""nodeType\"": \""string\"",\n \""name\"": \""string\"",\n \""current_pulse_instance_id\"": [\n 0\n ],\n \""previous_pulse_instance_id\"": [\n 0\n ]\n },\n \""question\"": \""how r u?\""\n}'```""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690870773.956639"", ""content"": ""i got the tracking id..when i query by trackingid i got\n```\t\nResponse body\nDownload\n{\n \""success\"": true,\n \""error\"": null,\n \""data\"": {\n \""trackingId\"": \""f2895b69-7868-428f-b6b9-92d9cc4de1cf\"",\n \""status\"": {\n \""status\"": \""SUCCESS\""\n },\n \""result\"": \""Patience, young grasshopper! The feature you seek is still brewing in the cauldron of creativity. Return in due time, and it shall reveal its magic to you!\""\n }\n}```""}, {""user"": ""svummidi"", ""timestamp"": ""1690870822.932709"", ""content"": ""Yes, it is working. But If I submit request using estimation service. I suspect it is going to a different cluster.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690870836.430279"", ""content"": ""ohhh..""}, {""user"": ""svummidi"", ""timestamp"": ""1690870885.454529"", ""content"": ""Just now i started a bounce on estimation service but not sure if it helps or not.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690870933.795299"", ""content"": ""do you have payload of estimation service - sample curl ?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690871038.367899"", ""content"": ""in estimation service askai pecification client is not sync with the latest askai server""}, {""user"": ""svummidi"", ""timestamp"": ""1690871041.954049"", ""content"": ""```curl -X 'POST' \\\n '' \\\n -H 'accept: application/json' \\\n -H 'Authorization: Bearer eyJraWQiOiJHUW9JcTh4Z3U5NFdGWCtyQ1BTcGJleHFwOGl6MmY5Y2diUUJVWjdpaDZVPSIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiI2bnU1Y2k1c2h1dXE3NjNtMG9va3QyZjVzNSIsInRva2VuX3VzZSI6ImFjY2VzcyIsInNjb3BlIjoiYXBpLWNhbGxzXC9yZXRyaWV2ZSBhcGktY2FsbHNcL3VwZGF0ZSBhcGktY2FsbHNcL2NyZWF0ZSBhcGktY2FsbHNcL2RlbGV0ZSIsImF1dGhfdGltZSI6MTY5MDg2ODAwMiwiaXNzIjoiaHR0cHM6XC9cL2NvZ25pdG8taWRwLnVzLXdlc3QtMi5hbWF6b25hd3MuY29tXC91cy13ZXN0LTJfWWZPU2oyNEdZIiwiZXhwIjoxNjkwODc1MjAyLCJpYXQiOjE2OTA4NjgwMDIsInZlcnNpb24iOjIsImp0aSI6IjdlZDQ5OGE0LWVkOTQtNDI0ZS1hZDVhLThkYTU3MDYzYjlhZSIsImNsaWVudF9pZCI6IjZudTVjaTVzaHV1cTc2M20wb29rdDJmNXM1In0.bHFUWS6H0Twsug8c5y93wWW_IYT_aRkPS4zxd14UqhaG1VSd_-zgskVWobwNMtQ3AgmpNLyny6HkWV6FTys9IXmbq1qd8nIg88YhqePf08WzDAoJbsFDhuPvNugrwASJwmFub1edKcLr8SmzUBWpUbFX2BDZ3iUVOo1BBLKN6xfzBuoXe2G1L6GJFUhWh3t_cUcsCFI-De9HVinjRRXthtr-qVGvjyE5L27aPOxhctUYfiWSuIJPciZQTP8rgzbMsXk1QeAxdXRTrtlF9d8LG4rZLEicEgWEOMYpEZgwJXDUzlmHo8G5uFVCy4fYEOWCQB_KFJb7cgYgjLyWZV51cg' \\\n -H 'Content-Type: application/json' \\\n -d '{\n \""estimationNode\"": {\n \""id\"": 22906,\n \""name\"": \""string\"",\n \""nodeType\"": \""project\""\n },\n \""userId\"": 0,\n \""questionText\"": \""What is the overall summary?\""\n}'```\n""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690871053.719699"", ""content"": ""Let me trigger a build of estimation service and then deploy""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690871112.587069"", ""content"": ""Is this the same error you were seeing\n""}, {""user"": ""svummidi"", ""timestamp"": ""1690871422.322769"", ""content"": ""Probably it is showing some old errors, I don\u2019t see any log with the task ID.""}, {""user"": ""svummidi"", ""timestamp"": ""1690871503.349249"", ""content"": ""If we use swagger to askAI it is showing some logs with the task ID.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690871716.469559"", ""content"": ""Quick call ?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690871730.507439"", ""content"": ""i see it as getting response""}, {""user"": ""svummidi"", ""timestamp"": ""1690871742.081119"", ""content"": ""sure.""}]" "1690555192.578209 ","[{""user"": ""aganivada"", ""timestamp"": ""1690555192.578209"", ""content"": ""<@U026PMDB1ND> <@U04JT69T00K> there was an issue with job-manager today it restarted automatically because of high CPU and picked up latest image, since this is python and we didn't have a automatic version update so the latest code got picked up which was expecting a env-var for microsoft account refresh. We didnt want to deploy to prod from develop branch due to all the changes happening so we updated main branch and deployed, change will be pushed to develop . now the job-manager is stable, there is still an issue with a data job I will initiate a separate thread on it.\n\nWhile debugging we were checking why would job-manager have high CPU, turns out it is publicly available on port 8080 on http.\n\n```anil_ganivada@ip-192-168-1-120 ~ % curl \n{\""detail\"":\""Not Found\""}%```\nand this node is being attacked every once in a while in prod and other env's . from CDK looks like the main difference is this stack has `apigw_enabled=make_static_param(fixed_value=False),` similar to UI stack so we seem to be allocating a public non internal domain in load-balancer. Now that we have ability to configure VPN validation I was thinking we can set `apigw_enabled` to true and let authorizer do the validation like rest of the backend services.""}, {""user"": ""askumar"", ""timestamp"": ""1690557222.294019"", ""content"": ""Calls log : similar reported by someone\n/laravel/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php\n\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690563908.293329"", ""content"": ""<@U02BV2DGUKC> - is this taken care of or do we need to do more to secure this?""}, {""user"": ""aganivada"", ""timestamp"": ""1690564019.264239"", ""content"": ""It is still open to outside Rama, if it looks ok we can turn on api-gw for job executor and enable vpn protection first we can check in int and then deploy to prod""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690564045.201019"", ""content"": ""Let's do this as soon as we can""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690564078.436449"", ""content"": ""I don't think this should be any different from any other service""}, {""user"": ""aganivada"", ""timestamp"": ""1690564174.046729"", ""content"": ""Agreed, not sure why we decided to disable api-gw initially though. Might be a miss""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690564184.797269"", ""content"": ""Yeah""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690564227.457089"", ""content"": ""Do we even know which jobs run today and if we really make use of this service?""}, {""user"": ""aganivada"", ""timestamp"": ""1690564309.674099"", ""content"": ""We are using for auth0 backups which is critical, other than that we are using for systest related cleanup and account refresh. <@U03KLHDKL1H> can you confirm if there are any other jobs?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690564350.184289"", ""content"": ""Do we know if the auth0 backups are running successfully?""}, {""user"": ""aganivada"", ""timestamp"": ""1690564375.041739"", ""content"": ""Yes <@U026PMDB1ND> I can see daily backups made to s3""}, {""user"": ""askumar"", ""timestamp"": ""1690564846.128829"", ""content"": ""Yeah <@U02BV2DGUKC> only these jobs :\nDATA_CLEANUP_JOB,\nAUTH0_BACKUP_JOB,\nMICROSOFT_ACCOUNT_REFRESHER_JOB""}, {""user"": ""askumar"", ""timestamp"": ""1690564956.665729"", ""content"": ""Auth0 Back up and Refresher are working okay.\n\nThe Data clean up is expecting some 'imply_password' to execute and it fails""}, {""user"": ""aganivada"", ""timestamp"": ""1692162774.633499"", ""content"": "" job-manager access issue is fixed in all 3 environments we should not see random get calls reaching job-manager service now""}]" "1687958351.883969 ","[{""user"": ""askumar"", ""timestamp"": ""1687958351.883969"", ""content"": ""<@U026PMDB1ND> <@U02BV2DGUKC>\nShould we now remove the RDS instances in INT, since all the service schemas have been moved to Aurora cluster?\n\ncc ""}, {""user"": ""aganivada"", ""timestamp"": ""1687959504.348919"", ""content"": ""<@U03KLHDKL1H> can we give few days for commons before deleting completely? For other dbs do we have backups in s3?""}, {""user"": ""askumar"", ""timestamp"": ""1687959725.130439"", ""content"": ""Sure <@U02BV2DGUKC> will keep them over next week.\nWe do have snapshot of the DB from which they can be restored.\nWill take backup as well to S3.""}, {""user"": ""aganivada"", ""timestamp"": ""1687960414.802339"", ""content"": ""BTW <@U03KLHDKL1H> we may not need all the backups just the latest one or might be sufficient""}, {""user"": ""askumar"", ""timestamp"": ""1687960473.760079"", ""content"": ""Yeah, latest backup of all RDS instances data.""}]" "1681206465.460379 ","[{""user"": ""aganivada"", ""timestamp"": ""1681206465.460379"", ""content"": ""<@U0431DZTPJM> can we take a look at ? I think this might be a small fix we need to lower case before comparison?""}, {""user"": ""ppant"", ""timestamp"": ""1681206532.024669"", ""content"": ""Sure <@U02BV2DGUKC>, will add `equalsIgnoreCase` check as part of refactoring""}, {""user"": ""ppant"", ""timestamp"": ""1681234420.455389"", ""content"": ""<@U02BV2DGUKC> Right now core data\u2019s `/publisher/all/{tenantId}` API is also case sensitive for domain in query filter. A easy fix is to convert the domain to lower case from tenancy and pass it to core. We are using `domains.domainName=in=()` this query filter so it does not ignore case""}, {""user"": ""aganivada"", ""timestamp"": ""1681234920.743829"", ""content"": ""ok when we store are we converting case?""}, {""user"": ""ppant"", ""timestamp"": ""1681261992.934269"", ""content"": ""Nope, no conversion to lower case are done\n```public class CoreDataMapperImpl implements CoreDataMapper {\n\n @Override\n public Publisher toEntity(PublisherCreationRequestDTO publisherCreationDTO) {\n if ( publisherCreationDTO == null ) {\n return null;\n }\n\n Publisher publisher = new Publisher();\n\n publisher.setName( publisherCreationDTO.getName() );```""}, {""user"": ""aganivada"", ""timestamp"": ""1681270362.008309"", ""content"": ""ok so we need rsql ignorecase? ""}, {""user"": ""aganivada"", ""timestamp"": ""1681270437.409389"", ""content"": ""may be we could use like operator?""}]" "1685708567.882089 ","[{""user"": ""ppant"", ""timestamp"": ""1685708567.882089"", ""content"": ""<@U026PMDB1ND> By default, the command `npx aws-cdk@2.x init app --language typescript` to install CDK v2 installs version 2.79.1 but some core-stacks are deployed with CDK v2.82.0 which is causing version mismatch issues. Maybe this is happening because npx version varies from people to people. This can be fixed by replacing `alias cdk=\""npx aws-cdk@2.x\""` with `alias cdk=\""npx aws-cdk@2.8.0\""` in common-core.sh""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685723971.275169"", ""content"": ""<@U0431DZTPJM> - how do the version mismatch issues manifest? What happens in deployment because of the mismatch?""}, {""user"": ""ppant"", ""timestamp"": ""1685724021.931409"", ""content"": ""<@U026PMDB1ND> Synth and deployment fails with this message\n```This CDK CLI is not compatible with the CDK library used by your application. Please upgrade the CLI to the latest version.\n(Cloud assembly schema version mismatch: Maximum schema version supported is 31.0.0, but found 32.0.0)\n[22:10:14] Error: This CDK CLI is not compatible with the CDK library used by your application. Please upgrade the CLI to the latest version.```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685724112.171119"", ""content"": ""Other than the suggested option (of fixing the version at a number), how does one get out of this?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685724148.976919"", ""content"": ""What does one need to do in order to \""upgrade the CLI to the latest version\""?""}, {""user"": ""ppant"", ""timestamp"": ""1685724195.827379"", ""content"": ""But if I explicitly set the version == 2.82.0 in common script, then it passes. Also. I checked what version of npx was installed in different machine. Ashwani has 9.x, Prashant 6.x and I had 8.x. So I guess if we mention cdk==2.x, then according to npx version different cdk version was getting installed. And somehow someone had 2.8 and deployed s3 stack and after which people with < 2.8 are getting this error""}, {""user"": ""ppant"", ""timestamp"": ""1685724293.980649"", ""content"": ""> What does one need to do in order to \u201cupgrade the CLI to the latest version\u201d?\nNot sure about this. Since we are installing cdk v2 via npx, I don\u2019t think we can update this after later on. All I did on my side was mention cdk==2.8.0 in the common script and that fixed it. Otherwise my cdk v2 was defaulting to 2.7""}, {""user"": ""ppant"", ""timestamp"": ""1685724507.037039"", ""content"": ""On a side note, I was able to deploy the app services fine. Faced this for core stack only""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685724559.799269"", ""content"": ""Interesting. Do you have the full output of the synth that failed.""}, {""user"": ""ppant"", ""timestamp"": ""1685724582.813269"", ""content"": ""```This CDK CLI is not compatible with the CDK library used by your application. Please upgrade the CLI to the latest version.\n(Cloud assembly schema version mismatch: Maximum schema version supported is 31.0.0, but found 32.0.0)\n[22:10:14] Error: This CDK CLI is not compatible with the CDK library used by your application. Please upgrade the CLI to the latest version.\n(Cloud assembly schema version mismatch: Maximum schema version supported is 31.0.0, but found 32.0.0)\n at createAssembly (/Users/prabhu/.npm/_npx/b2664a9b839d1ca8/node_modules/aws-cdk/lib/index.js:399:49964)\n at execProgram (/Users/prabhu/.npm/_npx/b2664a9b839d1ca8/node_modules/aws-cdk/lib/index.js:399:49318)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async CloudExecutable.synthesizer (/Users/prabhu/.npm/_npx/b2664a9b839d1ca8/node_modules/aws-cdk/lib/index.js:455:51119)\n at async CloudExecutable.doSynthesize (/Users/prabhu/.npm/_npx/b2664a9b839d1ca8/node_modules/aws-cdk/lib/index.js:399:41448)\n at async CloudExecutable.synthesize (/Users/prabhu/.npm/_npx/b2664a9b839d1ca8/node_modules/aws-cdk/lib/index.js:399:41241)\n at async CdkToolkit.selectStacksForDiff (/Users/prabhu/.npm/_npx/b2664a9b839d1ca8/node_modules/aws-cdk/lib/index.js:402:3030)\n at async CdkToolkit.synth (/Users/prabhu/.npm/_npx/b2664a9b839d1ca8/node_modules/aws-cdk/lib/index.js:402:537)\n at async exec4 (/Users/prabhu/.npm/_npx/b2664a9b839d1ca8/node_modules/aws-cdk/lib/index.js:455:51984)\n~/work/base-infra/core/scripts```""}, {""user"": ""ppant"", ""timestamp"": ""1685724625.994449"", ""content"": ""I ran this command btw `./cdkh.sh synth int ppant s3-axm-static-content-int`""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685725025.720389"", ""content"": ""Also, we needed npx and the alias option because we were having both cdk v1 and v2 for some time. Now that we moved to cdkv2 for all envs, we can change to install cdkv2 globally and update the instructions.""}, {""user"": ""ppant"", ""timestamp"": ""1685725091.331679"", ""content"": ""I will try with globally installed cdk==2.7 if this still happens""}, {""user"": ""ppant"", ""timestamp"": ""1685728231.384969"", ""content"": ""<@U026PMDB1ND> Works with global cdk. I removed the aliases from the common script and zsh env and installed cdk again from brew""}]" "1689846860.478529 ","[{""user"": ""aganivada"", ""timestamp"": ""1689846860.478529"", ""content"": ""<@U0431DZTPJM> QQ do you recollect when adding permissions to invitation why we choose only accountAdmin/productAdmin invites to have permissions but skip orgadmin? was it a safety check or is there some more critical reasoning? With CISO dashboard we need to have ability to also associate orgadmin invite with permissions so had to add line 312 in below screenshot""}, {""user"": ""ppant"", ""timestamp"": ""1689847003.385519"", ""content"": ""<@U02BV2DGUKC> At that time the requirement was for only these two roles to have permissions in invitations""}, {""user"": ""aganivada"", ""timestamp"": ""1689847123.198619"", ""content"": ""got it thank you <@U0431DZTPJM>, I think based on roles we have we can remove this check will test this with the condition for now and we can remove it later.""}, {""user"": ""ppant"", ""timestamp"": ""1689847192.453389"", ""content"": ""Sure <@U02BV2DGUKC>""}]" "1672813232.401649 ","[{""user"": ""pmangalapuri"", ""timestamp"": ""1672813232.401649"", ""content"": ""<@U02BV2DGUKC> is there a way to restrict docker image to be not overridden for a release version? Context being - when we rebased develop from main(though this is one-off manual error) we forgot to update to next dev version. I am assuming if we run develop build it will go override the release version docker image (assuming it is?). We will ensure this doesn't happen but can overriding the this release version be made little more restrictive?""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1672813847.713019"", ""content"": ""<@U02BV2DGUKC> please ignore, i see the develop branch always has -develop suffixed , so no issue here\naxm-ui-0.9.6-2-develop""}, {""user"": ""aganivada"", ""timestamp"": ""1672813859.033299"", ""content"": ""<@U02HCMTQU3W> the version of image to be built today is coming from projects metadata for UI I guess it is through a json file for most of other projects it comes through pom.xml and we assume that everyone is using gitflow. usually gitflow automatically handles this situation. Did we run into this since we have to update version manually for UI?""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1672814415.535679"", ""content"": ""Yes we do it manually""}]" "1676365231.962859 ","[{""user"": ""mnirmal"", ""timestamp"": ""1676365231.962859"", ""content"": ""```update svc_pulse_manager_default_axm.pulse_survey_user_instance set state='DELETED' where pulse_survey_instance_id=7846```\n```update svc_pulse_manager_default_axm.pulse_survey_question_response set state='DELETED' where \nid in (select psqr.id from svc_pulse_manager_default_axm.pulse_survey_question_response psqr \n\t join svc_pulse_manager_default_axm.pulse_survey_user_instance psui on psqr.pulse_survey_user_instance_id = \n\t psui.id where psui.pulse_survey_instance_id=7846);```\n<@U02D4DUKDQC> when we ran the query to delete the instance last week on PROD, we missed deleting the user instances and responses.\n\nCan you please review these queries once, so <@U02BV2DGUKC> and execute them on PROD?""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1676368545.022409"", ""content"": ""Looks good <@U02SF36PVKL>""}, {""user"": ""mnirmal"", ""timestamp"": ""1676368598.827589"", ""content"": ""<@U02BV2DGUKC> can we run the queries?""}, {""user"": ""aganivada"", ""timestamp"": ""1676368782.731569"", ""content"": ""sure <@U02SF36PVKL>, I am in a call now will run these after the call""}, {""user"": ""aganivada"", ""timestamp"": ""1676369696.430969"", ""content"": ""<@U02SF36PVKL> this is done\n\n> ```update svc_pulse_manager_default_axm.pulse_survey_user_instance set state='DELETED' where pulse_survey_instance_id=7846```\n> \n2 records updated\n\n> ```update svc_pulse_manager_default_axm.pulse_survey_question_response set state='DELETED' where \n> id in (select psqr.id from svc_pulse_manager_default_axm.pulse_survey_question_response psqr \n> \t join svc_pulse_manager_default_axm.pulse_survey_user_instance psui on psqr.pulse_survey_user_instance_id = \n> \t psui.id where psui.pulse_survey_instance_id=7846);```\n> \nNone""}]" "1686633495.302029 ","[{""user"": ""hchintamreddy"", ""timestamp"": ""1686633495.302029"", ""content"": "" there are a few indexing failures for prod logs , these are logs from lambda I think""}, {""user"": ""askumar"", ""timestamp"": ""1686646448.520829"", ""content"": ""Looks like a common failure across the lambdas.\n<@U0431DZTPJM> is taking a look.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686674235.164369"", ""content"": ""Thanks for flagging this <@U02D4DUKDQC>\n\n<@U0431DZTPJM> - is this taken care of?""}, {""user"": ""askumar"", ""timestamp"": ""1686674661.170239"", ""content"": ""Prabhu raised an MR for this :\n""}]" "1689324865.208239 ","[{""user"": ""aganivada"", ""timestamp"": ""1689324865.208239"", ""content"": ""<@U04JT69T00K> does gitlab allow updating default maximum TTL for job beyond 1 hour? cc: <@U03NZ7Z52S2>""}, {""user"": ""pjha"", ""timestamp"": ""1689325039.838299"", ""content"": ""I think it allows to change the default TTL, let me check""}, {""user"": ""pjha"", ""timestamp"": ""1689325086.519469"", ""content"": """"}, {""user"": ""pjha"", ""timestamp"": ""1689325116.189739"", ""content"": ""we can modify the timeout here""}, {""user"": ""aganivada"", ""timestamp"": ""1689325126.992009"", ""content"": ""Cool thank you <@U04JT69T00K>""}]" "1679899671.105509 ","[{""user"": ""pkarthikeyan"", ""timestamp"": ""1679899671.105509"", ""content"": "" Perimeter 81 is not getting connected. (says reconnecting and does not connect). Can someone take a look?""}, {""user"": ""pkarthikeyan"", ""timestamp"": ""1679899797.181099"", ""content"": ""FYI: Also noticed that we are unable to connect to RDS when Perimeter 81 is connected. Would be nice to have this fixed. Not a priority though.""}, {""user"": ""ppant"", ""timestamp"": ""1679912514.660429"", ""content"": ""<@U03BPNY5AGM> Could you try now? I just tried and was able to connect to Perimeter81. Also I am able to connect to RDS while in VPN""}, {""user"": ""askumar"", ""timestamp"": ""1679916419.298439"", ""content"": ""Working now for me as well , looks like some transient issue only""}, {""user"": ""svummidi"", ""timestamp"": ""1679939913.245509"", ""content"": ""<@U0431DZTPJM> - During last week not able to use sshuttle to connect to Prod Postgres from VPN. Is it expected to work for all environments? Anything we fixed to make it working?""}, {""user"": ""ppant"", ""timestamp"": ""1679965674.314959"", ""content"": ""Not sure about env specific sshuttle restrictions <@U040RCBPBEC>, maybe <@U02BV2DGUKC> will know more on this. For me the connection to int RDS instance from VPN worked without any hiccups""}, {""user"": ""aganivada"", ""timestamp"": ""1679973532.120119"", ""content"": ""> Perimeter 81 is not getting connected\n<@U03BPNY5AGM> looks like there was a perimeter81 outage yesterday ""}, {""user"": ""aganivada"", ""timestamp"": ""1679973747.516439"", ""content"": ""> During last week not able to use sshuttle to connect to Prod Postgres from VPN. Is it expected to work for all environments? Anything we fixed to make it working?\n<@U040RCBPBEC> Our VPN currently gives private access to int and stage env, for prod access we don't need VPN yet. sshuttle should work seamlessly. However, if there is any outage of VPN and if we are connected to VPN during this time it is likely that the connection might break.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679974797.668599"", ""content"": ""<@U02BV2DGUKC> - lots of tests seem to be failing even today. Maybe they are still having intermittent issues..""}, {""user"": ""aganivada"", ""timestamp"": ""1679979918.379799"", ""content"": ""sure<@U026PMDB1ND> test failure seem to be because ST changes for vpn haven't been merged to develop yet. for now I removed the check on tenancy so tests should succeed I can enable it when I am testing the vpn branch.""}]" "1680286679.473949 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1680286679.473949"", ""content"": ""<@U02BV2DGUKC> - On the call with LVSands for SSO - there was an ask that if provide just in time provisioning - where user is assigned a group /role in auth0 - automatically he should be able to login and view dashboard as collabarator with out orgadmin adding him as collabarator in our UI - can you add a backlog - we can syncup and explore on this ""}, {""user"": ""aganivada"", ""timestamp"": ""1680341791.609259"", ""content"": ""Sure thank you <@U0336QZAF98>, doesn't seem like a straight forward thing from auth0. Let's sync up on this next week will add a platform ticket.""}]" "1680797669.930769 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1680797669.930769"", ""content"": ""<@U026PMDB1ND> <@U02BV2DGUKC> - Below are the projects whihc i have worked - where in build process - step 1) tests where everything runs and again one step deploy - where again it runs everything - i have seen this pattern in multiple places -may be we can do an excercise where we go over all the projects and fix if not so\n```vms-estimation-service\nplatform-services-notification\nvms-services-ingestion\nvms-services-score-provider\nplatform-services-tenant\npulse-manager```""}, {""user"": ""aganivada"", ""timestamp"": ""1680803536.312489"", ""content"": ""Probably needs .gitlab.ci.yml update to align with other services.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681350322.077479"", ""content"": ""I was trying to fix these repos. Most of them are done. I was also trying to fix the code coverage badge issue in the process.\n\nplatform-services-notfication is dumping a lot of text on to the console. Turns out openapi generator is the issue because we have `<verbose>true</verbose>`\n\nAlso turned of hibernate.show-sql.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681358698.375309"", ""content"": ""<@U028EDANJM9> - also removed verbose true in Discovery and got its coverage badge to show up""}, {""user"": ""gshenoy"", ""timestamp"": ""1681358767.869649"", ""content"": ""Thank you <@U026PMDB1ND>""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1681359817.190539"", ""content"": ""Thank you <@U026PMDB1ND>""}]" "1692268179.127169 ","[{""user"": ""aganivada"", ""timestamp"": ""1692268179.127169"", ""content"": ""<@U03KLHDKL1H> does snowflake support read only access? <@U02TVMF3CR4> wanted access to env to check the schema""}, {""user"": ""askumar"", ""timestamp"": ""1692268216.206139"", ""content"": ""Yes <@U02BV2DGUKC> for Product we have read only access""}, {""user"": ""sjain"", ""timestamp"": ""1692268251.310509"", ""content"": ""Will it allow writing new queries for new chart ?""}, {""user"": ""aganivada"", ""timestamp"": ""1692268252.975429"", ""content"": ""Ok can we add a user for Sanket?""}, {""user"": ""askumar"", ""timestamp"": ""1692268442.249909"", ""content"": ""Sure let me add one more user.\n<@U02TVMF3CR4> yes we can add more queries""}, {""user"": ""aganivada"", ""timestamp"": ""1692268572.385159"", ""content"": ""<@U03KLHDKL1H> is there anyway we can restrict env access from snowflake? Like restrict access to production data?""}, {""user"": ""askumar"", ""timestamp"": ""1692269189.367959"", ""content"": ""<@U02BV2DGUKC> we have permission on table level and data from both INT and PROD is going to same tables, so can't have env level access.""}, {""user"": ""askumar"", ""timestamp"": ""1692269248.804139"", ""content"": ""Added User : Sanket with view only access, Also provided a replica of dashboard which can be edited for testing new queries.""}, {""user"": ""aganivada"", ""timestamp"": ""1692269375.867029"", ""content"": ""<@U02TVMF3CR4> please let us know once you are done, we have to limit access to production data for compliance reasons and since we don't have a way to control it in snowflake yet we have to restrict users who have access to snowflake.""}, {""user"": ""sjain"", ""timestamp"": ""1692606516.189389"", ""content"": ""Hey <@U02BV2DGUKC>, we were able to put some new tiles on the dashboard. Thanks to <@U03KLHDKL1H> for updating the prod dashboard.\n\nWe have asked Aryan to take. a look once.\nI don\u2019t need the dashboard access unless Aryan would need to create/modify any of the query.""}, {""user"": ""aganivada"", ""timestamp"": ""1692615839.375939"", ""content"": ""sure thank you <@U02TVMF3CR4> & <@U03KLHDKL1H>.\n\n <@U03KLHDKL1H> can we revert Sanket's access on snowflake. We can add it back if required.""}, {""user"": ""askumar"", ""timestamp"": ""1692616025.185019"", ""content"": ""sure <@U02BV2DGUKC>, revoked the access from snowflake.""}]" "1689867676.716119 ","[{""user"": ""aganivada"", ""timestamp"": ""1689867676.716119"", ""content"": ""<@U04JT69T00K> looks like metrics-manager service doesn't have VPN enabled, can we check and update?""}, {""user"": ""pjha"", ""timestamp"": ""1689870613.953459"", ""content"": ""Deployed the changes for the metrics service.\nActually this service was recently merged to develop thats why I missed to update it.""}]" "1680112156.981489 ","[{""user"": ""pjha"", ""timestamp"": ""1680112156.981489"", ""content"": ""<@U026PMDB1ND> <@U02BV2DGUKC> Please review PR for replacing ECS ip with alb endpoint in route53 ""}, {""user"": ""aganivada"", ""timestamp"": ""1680148813.086319"", ""content"": ""<@U04JT69T00K> can we test this with some service (tenancy?) in int before we merge?""}, {""user"": ""pjha"", ""timestamp"": ""1680149365.940159"", ""content"": ""Sure""}]" "1686785392.370739 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1686785392.370739"", ""content"": "" - has we asked for any change recently in how Logz handles our logs? Regex is not working for the most part on Stage and Prod accounts, but seems to work for INT, which is very strange.\n\nWonder if anyone asked them to change anything and if this is a main/side-effect of that change.""}, {""user"": ""aganivada"", ""timestamp"": ""1686797655.125509"", ""content"": ""not sure <@U026PMDB1ND> I haven't raised any logz ticket recently, there was one mapping issue related to mdc.tenantId but that change seemed harmless.""}]" "1693993596.265039 ","[{""user"": ""aganivada"", ""timestamp"": ""1693993596.265039"", ""content"": ""<@U02HCMTQU3W> me and <@U03NZ7Z52S2> were debugging it looks like in admins page when we click on re-send we seem to be sending invitation but not triggering delete invitation API from dashboard-app is this expected? I was under the assumption that we first trigger delete invitation and then trigger invitation flow during re-send. cc: <@U02GC8SE18V>""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1694076157.966949"", ""content"": ""<@U02GC8SE18V> Should we do this from ui or can apps do it?""}, {""user"": ""rvaidya"", ""timestamp"": ""1694076540.257119"", ""content"": ""<@U02HCMTQU3W> i wasnt tracking it. But <@U02BV2DGUKC> can this not be done as part of tenancy invite API itself ?""}, {""user"": ""aganivada"", ""timestamp"": ""1694076984.212979"", ""content"": ""hmmm.......<@U02GC8SE18V>/<@U02HCMTQU3W> we already have a delete invitation API from tenancy and it seems its also available through dashboard-app can we use it? Handling through invite API might still need apps and UI changes to add additional parameter indicating that the operation is a re-send""}]" "1683192179.672009 ","[{""user"": ""ppant"", ""timestamp"": ""1683192179.672009"", ""content"": "" Need some opinions and help on why OpenApi\u2019s ApiClient, when made a Spring component, fails to picks up a bean from plat-commons.\n\nContext: For multi instance deployments, we are trying to customise the API Client that is generated by OpenApi codegen so that we can route requests to a custom endpoint for services depending on the tenant ID in the ThreadContext. For this, we did three things -\n1. In `plat-libs-commons`, we created a Spring Component called `AxmApiClient` which reads application.yml for reading endpoint configs and returns the basePath according to the tenant ID in the ThreadContext.\n2. Next, we added the webclient mustache template of openapi-codegen () as well as its plugin in tenancy\u2019s specification\u2019s pom.xml. Now, we are able to modify the generated code of `ApiClient.java` by modifying the mustache template.\n3. We injected the bean of `AxmApiClient` in `ApiClient` (openapi generated one) using @Autowired and modified the `getBasePath` to call `axmApiClient.getBasePath(str)` instead for routing logic.\nThe problem that\u2019s coming is that AxmApiClient comes as null in ApiClient when we run the application. Going by the debug logs, it seemed `apiClient` bean was being created before `axmApiClient` so that could have been the cause but even using @DependsOn(\u201caxmApiClient\u201d) on ApiClient does not fix the ordering. Here are the MRs for this -\n and \n\nThe modified APIClient works when it calls static methods of AxmApiClient (both are not beans) but this will prevent us from reading application.yml. Rather, we can try reading a new JSON file for routing or read it from AWS Paramter Store.""}, {""user"": ""aganivada"", ""timestamp"": ""1683193045.297399"", ""content"": ""<@U0431DZTPJM> in conclusion can we say that root cause is autowiring of axmApiClient is not working in apiClient?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1683193362.636439"", ""content"": ""Instead of @autowiring here - you can have that a setter/getter in ApiClient- let the calling guy - which will typically be a component or service - do a set of AxmApiClient - bcoz openapi generated clients are typically pure java (no spring bean)""}, {""user"": ""aganivada"", ""timestamp"": ""1683193834.260149"", ""content"": ""<@U0336QZAF98> that is an alternative but before going there we wanted to check if we can do something at library so existing code doesn't need change.""}, {""user"": ""aganivada"", ""timestamp"": ""1683193858.135589"", ""content"": ""Prabhu is seeing that the bean is getting initiazed but somehow the order is screwed""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1683194200.913309"", ""content"": ""```In MR i see - So ApiClient is component. Are we creating via new or via injection ?\n inflightApiClient =\n new ApiClient(webClient).setBasePath(serverLocation + BASE_PATH);```""}, {""user"": ""aganivada"", ""timestamp"": ""1683194502.560159"", ""content"": ""hmmm it should be bean not with new constructor, Prabhu is checking this""}, {""user"": ""ppant"", ""timestamp"": ""1683199578.978129"", ""content"": ""Thanks <@U02BV2DGUKC> and <@U0336QZAF98>, got this working now. The problem was that, as <@U0336QZAF98> pointed out, we were creating the ApiClient via new keyword as we had to pass the webclient here. This was common across all clients so I did not change it here. Also using @Autowired for injection was resulting in null, when we updated it to constructor injection, it worked""}]" "1677351784.805629 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1677351784.805629"", ""content"": ""<@U03NZ7Z52S2> <@U03DHUAJVMK> - a number of bugs opened in Feb don't have the Testing Complexity field populated: \n\nPlease see if you can update the same""}, {""user"": ""bganganna"", ""timestamp"": ""1677385165.875679"", ""content"": ""Sure <@U026PMDB1ND>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677519568.699599"", ""content"": ""We added five more to get to a half-century of bugs without the field populated :slightly_smiling_face:""}, {""user"": ""bganganna"", ""timestamp"": ""1677556344.216179"", ""content"": ""I will update the tickets which were created before adding this field <@U026PMDB1ND> . Can we make this field as mandatory so we ll not miss to update ?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677604791.455819"", ""content"": ""As long as there's a default value, I guess we can make it mandatory""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677604817.809499"", ""content"": ""<@U03DHUAJVMK> - do you see any issue in making the field mandatory?""}, {""user"": ""araman"", ""timestamp"": ""1677605816.985229"", ""content"": ""Nope, let me do that""}]" "1691467589.197369 ","[{""user"": ""aganivada"", ""timestamp"": ""1691467589.197369"", ""content"": ""<@U03KLHDKL1H> moving the thread here. Does this mean it got updated later?""}, {""user"": ""askumar"", ""timestamp"": ""1691467716.517349"", ""content"": ""Will need to check.\nBut weirdly enough only score provider is false, rest are good in main, although they were part of same change.""}, {""user"": ""aganivada"", ""timestamp"": ""1691468117.054389"", ""content"": ""may be we updated manually later on?""}, {""user"": ""askumar"", ""timestamp"": ""1691468124.445849"", ""content"": ""in develop branch it shows 20th June and value is false.\nBut this commit was merged on 7th July, so it should have overridden it.""}, {""user"": ""askumar"", ""timestamp"": ""1691468160.766129"", ""content"": ""Yeah it could be that it was updated later directly.""}]" "1676652075.360069 ","[{""user"": ""araman"", ""timestamp"": ""1676652075.360069"", ""content"": ""<@U02BV2DGUKC> I have raised questions and awaiting replies on few changes. Are there any fixes expected to be pushed today?""}, {""user"": ""aganivada"", ""timestamp"": ""1676652243.280319"", ""content"": ""nothing that I am aware of <@U03DHUAJVMK>, EA bulk upload has already been deployed. lets wait for confirmation from Product""}, {""user"": ""araman"", ""timestamp"": ""1676652289.336819"", ""content"": ""Thank you!""}]" "1681314704.421059 ","[{""user"": ""rvaidya"", ""timestamp"": ""1681314704.421059"", ""content"": ""<@U02BV2DGUKC> Any idea about this error from vendorapp in INT :\norg.springframework.cloud.stream.binder.BinderException: Exception thrown while building outbound endpoint\n\tat org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindProducer(AbstractMessageChannelBinder.java:251)\n\tat org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindProducer(AbstractMessageChannelBinder.java:92)\n\tat org.springframework.cloud.stream.binder.AbstractBinder.bindProducer(AbstractBinder.java:152)\n\tat org.springframework.cloud.stream.binding.BindingService.lambda$rescheduleProducerBinding$4(BindingService.java:351)\n\tat org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: com.amazonaws.services.kinesis.model.AmazonKinesisException: User: arn:aws:sts::433798924509:assumed-role/ecs-task-role-name-int/b8a469c98ffa41c892b6bd96a68be61e is not authorized to perform: kinesis:ListShards on resource: arn:aws:kinesis:us-west-2:433798924509:stream/springCloudBus because no identity-based policy allows the kinesis:ListShards action (Service: AmazonKinesis; Status Code: 400; Error Code: AccessDeniedException; Request ID: d92eb5d1-b8c6-ca85-859e-61cfac7b31d4; Proxy: null)""}, {""user"": ""aganivada"", ""timestamp"": ""1681315167.243899"", ""content"": ""<@U02GC8SE18V> do we have ECS IAM role permissions to access this Kinesis instance?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1681319398.921899"", ""content"": ""typically all micro services has full permissions to access kinesis if deployed via cdk..""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1681319715.478199"", ""content"": ""```The name of kinesis stream should start with environment - int-*\n \n\n self.ecs_task_role.add_to_policy(\n statement=iam.PolicyStatement(\n sid=\""KinesisAccess\"",\n effect=iam.Effect.ALLOW,\n actions=[\n \""kinesis:*\"",\n ],\n resources=[\""arn:aws:kinesis:*:%s:stream/%s*\"" % (envw.account_details.account, envw.env)],\n )\n )```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681320136.843769"", ""content"": ""``` {\n \""Action\"": \""kinesis:*\"",\n \""Resource\"": \""arn:aws:kinesis:*:433798924509:stream/int*\"",\n \""Effect\"": \""Allow\"",\n \""Sid\"": \""KinesisAccess\""\n },```\nThis policy exists in `ecs-task-role-name-int`""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681320196.974659"", ""content"": ""In arn:aws:kinesis:us-west-2:433798924509:stream/springCloudBus, \""springCloudBus\"" doesn't start with \""int-\""""}, {""user"": ""aganivada"", ""timestamp"": ""1681360366.307889"", ""content"": ""it seems service is not picking up the kinesis we configured for cloud config events""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681360498.359349"", ""content"": ""How come only this service has a problem. It should be the case with others as well, right?""}, {""user"": ""rvaidya"", ""timestamp"": ""1681360542.659829"", ""content"": ""We have 2 configs : application.yaml and application-int.yaml in the dev config\n\n\n\nThis is the only diff i could see.\n<@U026PMDB1ND> <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1681360621.380299"", ""content"": ""<@U02GC8SE18V> can I remove application-int.yml and update application.yml with\n\n> ```cloud:\n> bus:\n> enabled: true\n> destination: ${CONFIG_EVENT_STREAM:axm-config-events}\n> stream:\n> bindings:\n> springCloudBusInput:\n> destination: ${CONFIG_EVENT_STREAM:axm-config-events}\n> \n> springCloudBusOutput:\n> destination: ${CONFIG_EVENT_STREAM:axm-config-events}\n> input:\n> #destination: test-config-events\n> group: config-event\n> content-type: application/json```\n> ""}, {""user"": ""rvaidya"", ""timestamp"": ""1681360640.852689"", ""content"": ""yes we can remove <@U02BV2DGUKC>, i can do that too\u2026lmk""}, {""user"": ""ppant"", ""timestamp"": ""1681360657.664139"", ""content"": ""<@U02GC8SE18V> Referring from tenancy\u2019s application.yml, `stream` tag should be inside `cloud`. For vendor dashboard, both cloud and stream are at the same level ""}, {""user"": ""aganivada"", ""timestamp"": ""1681360671.175209"", ""content"": ""sure please go ahead if application-int yml has no special configs we can remove it""}, {""user"": ""aganivada"", ""timestamp"": ""1681360708.773579"", ""content"": ""as Prabhu mentioned spring cloud integration needs a stream if stream is not passed through config then it pick the default SpringCloudBus""}, {""user"": ""rvaidya"", ""timestamp"": ""1681360717.346019"", ""content"": ""Thanks <@U0431DZTPJM>, i will fix that as well.\n\nHowever, the changes are getting picked from config \u2026so everything works as expected except this error every 30 sec.""}, {""user"": ""aganivada"", ""timestamp"": ""1681360756.607459"", ""content"": ""yes <@U02GC8SE18V> this is because Stream by default looks for input and output params""}, {""user"": ""aganivada"", ""timestamp"": ""1681360804.508259"", ""content"": ""if not passed it will pick the default ones so with the config above we are connecting them to the kinesis stream which allows connection with ECS tasks""}, {""user"": ""rvaidya"", ""timestamp"": ""1681362478.182509"", ""content"": ""<@U02BV2DGUKC> <@U0336QZAF98> <@U026PMDB1ND> <@U0431DZTPJM> The issue got resolved when i removed *-int.yaml/s from the dev config.\nAlso cleaned up application.yaml for spring.cloud.bus property since it is coming via the jar/application yaml.\nThanks for looking into this.""}, {""user"": ""aganivada"", ""timestamp"": ""1681362527.390969"", ""content"": ""<@U02GC8SE18V> can we also check stage and production config folders""}, {""user"": ""rvaidya"", ""timestamp"": ""1681362593.007579"", ""content"": ""Just checked, it is fine for Prod config. <@U02BV2DGUKC>""}]" "1676612549.393719 ","[{""user"": ""aganivada"", ""timestamp"": ""1676612549.393719"", ""content"": ""<@U0431DZTPJM> added for customizing email templates during re-invite for next sprint, please check if you have bandwidth we can even take this up as stretch for this sprint.""}, {""user"": ""ppant"", ""timestamp"": ""1676612609.128899"", ""content"": ""Sure :+1: For this I am thinking if we pass a query param like `reinvite=true` and then while selecting template pick up the reinvite one?""}, {""user"": ""aganivada"", ""timestamp"": ""1676612725.524849"", ""content"": ""yeah please see if it is possible through auth0 otherwise we'd have to find a way to suppress it from auth0""}]" "1678145752.152829 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1678145752.152829"", ""content"": ""<@U02GC8SE18V> - when I refer to a model class elsewhere IntelliJ thinks that there's an error in my project (see attached) though compilation from the CLI goes through just fine.\n\nI know that you have done this extensively in Apps services. Is there a tip/trick to make IntelliJ behave correctly? Thanks!""}, {""user"": ""rvaidya"", ""timestamp"": ""1678156744.105069"", ""content"": ""I have lived with this error in intellij. If compilation goes fine, i wont worry.\nBtw this started coming after we did spring dep update.\n<@U026PMDB1ND>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678157301.669339"", ""content"": ""sadness :|""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678252755.467909"", ""content"": ""<@U02D4DUKDQC> - do you run into this or do you have a workaround?""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1678257471.457059"", ""content"": ""let me take a look <@U026PMDB1ND> from what I remember the correct way to do this to define it in requestBodies and refer it""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678302188.923279"", ""content"": ""Hmm ok""}]" "1676360439.718969 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1676360439.718969"", ""content"": ""If you are looking for a better terminal than iterm - give this a try ""}, {""user"": ""ppant"", ""timestamp"": ""1676360651.697679"", ""content"": ""I tried using this, was pretty good. But getting tmux into this is difficult""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1676360902.063079"", ""content"": """"}]" "1691663919.190519 ","[{""user"": ""askumar"", ""timestamp"": ""1691663919.190519"", ""content"": ""<@U026PMDB1ND> <@U02BV2DGUKC>\n\nRegarding configuring ILM policies, need guidance on below tasks :\n\u2022 Currently we have a single instance in INT, how should we test it out for our data and Indexes, since policies can be applied only on Cluster.\n\u2022 Also the indexes that are currently created have a Static name, should we also make them flexible and append timeStamp in index name like Index+Date.\nChanging the index to index+date pattern would help in easily deciding and applying policies on day level Indexes.\nThis way we can decide the indexes of last N days to kept in Hot storage and move them to Cold on day basis.\n\nAlso having the index with timestamp would easily help move data from Cold store to Hot, on demand by identifying easily from the name only.\n\nPlease let me know your thoughts on this.\nThanks\ncc ""}, {""user"": ""askumar"", ""timestamp"": ""1691664098.172769"", ""content"": ""Related doc : ""}, {""user"": ""aganivada"", ""timestamp"": ""1691665819.985659"", ""content"": ""> \u2022 Currently we have a single instance in INT, how should we test it out for our data and Indexes, since policies can be applied only on Cluster.\n<@U03KLHDKL1H> what will be the delta cost of enabling cluster mode in int/stage? if it is very costly may be we can we provision new cluster in int -> copy data over -> point audit log service in int to new cluster for testing and revert after enabling policies in prod.\n\n> \u2022 Also the indexes that are currently created have a Static name, should we also make them flexible and append timeStamp in index name like Index+Date.\n> \nif we are planning for daily backup then may be Index+Date or if it is weekly then Index+week_of_year""}, {""user"": ""askumar"", ""timestamp"": ""1691674455.101999"", ""content"": ""Thanks <@U02BV2DGUKC>\nAgreed data copy and testing with cluster could help in testing in INT.\n1. \nThe cost are as follows :\n\u2022 Prod 7 Nodes $490\n Stage 2 Nodes ~ $120\n\nIf we use same number of nodes as there are in Prod and turn 2 into ultra warm type for ilm policies to be applicable, we can get around $450.\nMinimum of 2 ultrawarm are required ($350)\n\n2. Need to decide on frequency of data availability daily/weekly indexes.""}, {""user"": ""aganivada"", ""timestamp"": ""1691680823.828909"", ""content"": ""do we have to use same nodes as that of prod for testing in stage? what is the minimum number of nodes required?""}, {""user"": ""askumar"", ""timestamp"": ""1691681600.694919"", ""content"": ""Actually <@U02BV2DGUKC> we could do it it less number of nodes as below :\n1. Master -1\n2. Data -1\n3. UltraWarm-2 (minimum 2 are required as per AWS , this is major cost $350, only two types of instances are available under ultrawarm, I am using lesser configuration one )\nSo minimum 4.\nI will try reducing from the created test cluster and see what minimum we can get to.""}, {""user"": ""aganivada"", ""timestamp"": ""1691682317.570609"", ""content"": ""after testing can we switch back to non-cluster mode in int/stage or do we have to continue with cluster mode in atleast one lower env?""}, {""user"": ""askumar"", ""timestamp"": ""1691683100.753929"", ""content"": ""Yeah, I think data can be copied over through script/manually to one instance, would review what will be required for this.\n\nThis should be doable.""}]" "1684257171.549539 ","[{""user"": ""aganivada"", ""timestamp"": ""1684257171.549539"", ""content"": """"}, {""user"": ""askumar"", ""timestamp"": ""1684258252.171549"", ""content"": ""<@U02BV2DGUKC> for data specifically we don't need to migrate, since we already have one month data in S3 that will get added when we add SnowPipe to new account.\n\nThe suggestion in this community post as well is also about getting data into AWS S3 and doing a copy into our table.""}, {""user"": ""aganivada"", ""timestamp"": ""1684258665.095079"", ""content"": ""ok so if we provision a new account do we have ability to export and import config/scripts you have written in snowflake?""}]" "1686240468.613329 ","[{""user"": ""aganivada"", ""timestamp"": ""1686240468.613329"", ""content"": ""<@U026PMDB1ND> for dynamodb backup we are planning to leverage point-in-time-recovery which provides 35 days of backup and backups are taken daily. There is just one line we need to add to CDK and deploy all the dynamodb instances. From pricing perspective dynamodb doesnt seem to be that heavy in worst case even if we have a backup of 1G per day (currently it is in KB's) for mothly costs we may have an overhead of $ 6""}, {""user"": ""aganivada"", ""timestamp"": ""1686240534.888499"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1686240566.151359"", ""content"": ""Will add a task to try to recover a table and record the steps.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686242432.031969"", ""content"": ""Thanks <@U02BV2DGUKC>""}]" "1688698250.047579 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1688698250.047579"", ""content"": ""<@U03KLHDKL1H> - in cdk json I made change to point to aurora cluster in int - but when I deploy it is pointing to old common RDS cluster - I have latest develop code - I am missing something here - should I need to use some specific branch ?""}, {""user"": ""askumar"", ""timestamp"": ""1688702092.647079"", ""content"": ""Which service <@U0336QZAF98>?""}, {""user"": ""askumar"", ""timestamp"": ""1688702284.710889"", ""content"": ""All the changes have been merged to develop, should not be such an issue.\nIf you are getting the correct PG_HOSTNAME in synth and incorrect while deploying then it might be some caching issue.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1688712513.068559"", ""content"": ""This is new service - askai - Its in branch feature/DAT-1062 - there is some problem in aliasing - where iot goes to common old rds instance instead of new aurora cluster""}, {""user"": ""askumar"", ""timestamp"": ""1688721211.039579"", ""content"": ""We were able to get it working by creating two environments, since the stack is deploying 2 services (askai, askai-celery) internally.""}, {""user"": ""aganivada"", ""timestamp"": ""1688723841.909419"", ""content"": ""<@U03KLHDKL1H> just wanted to check how is this different from regular service, can you add more details and may be commit if possible?""}, {""user"": ""askumar"", ""timestamp"": ""1688724275.258339"", ""content"": ""<@U02BV2DGUKC> This stack is slightly different. this stack is creating one environment and deploying 2 services separately as part of one stack.\nThese 2 services are independent and have their own environment.\n<@U0336QZAF98> if you could please elaborate more here, regarding the different implementation.\n\n\nThere were 2 issues :\n1. Since the stack is deploying 2 services, so firstly the migration tag was added for 'askai' and not for 'askai-celery'\n2. However even after adding the above flag, the CDK.json overrides were getting picked up when first service was being deployed i.e 'askai'\n3. However when stack was going to deploy 'askai-celery service', the global CDK.json was being picked up and not the overrides in the 'int'.\nSo to overcome this, we tried creating 2 cdk environment (one line change in Service stack) while deploying and then it worked fine.""}, {""user"": ""askumar"", ""timestamp"": ""1688724400.553539"", ""content"": ""There is no change in baseApp stack, Seshan has the change in his app stack , which he is using to deploy.""}, {""user"": ""aganivada"", ""timestamp"": ""1688724477.761649"", ""content"": ""Got it thank you <@U03KLHDKL1H>, <@U0336QZAF98> do the 2 services have their own versions or they use same version?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1688724622.123889"", ""content"": ""I have created a docker image - same image is published as askai-service and askai-celery - - By passing parameter to the startup script - it can either run as a webservice (8080) or celery background task (no alb , no gateway - just a daemon)""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1688724632.615449"", ""content"": ""Right now both use the same version from pom file""}]" "1686149934.127609 ","[{""user"": ""aganivada"", ""timestamp"": ""1686149934.127609"", ""content"": "" can anyone try deploying any service/core stack to int? I am getting s3 access denied error. I did deploy admin-gw this afternoon without any issues not sure what changed""}, {""user"": ""askumar"", ""timestamp"": ""1686149955.903879"", ""content"": ""sure <@U02BV2DGUKC>""}, {""user"": ""askumar"", ""timestamp"": ""1686149996.080619"", ""content"": ""let me deploy core data""}, {""user"": ""askumar"", ""timestamp"": ""1686150104.795129"", ""content"": ""Started deployment for core""}, {""user"": ""aganivada"", ""timestamp"": ""1686150110.045159"", ""content"": ""thank you Ashwani""}, {""user"": ""aganivada"", ""timestamp"": ""1686150334.583269"", ""content"": ""looks like it is working Ashwani""}, {""user"": ""aganivada"", ""timestamp"": ""1686150342.801439"", ""content"": ""probably my env got messed up""}, {""user"": ""aganivada"", ""timestamp"": ""1686150355.506389"", ""content"": ""let me restart and check""}, {""user"": ""askumar"", ""timestamp"": ""1686150359.113719"", ""content"": ""yes cloud formation seems to be triggered and update works""}, {""user"": ""askumar"", ""timestamp"": ""1686150362.025649"", ""content"": ""cool""}, {""user"": ""aganivada"", ""timestamp"": ""1686150412.039839"", ""content"": ""meanwhile can you deploy \""0.1.0-PLAT-2037-SNAPSHOT\"" of admin-gw once core is done just realised that the deployment didn't go through""}, {""user"": ""askumar"", ""timestamp"": ""1686150428.933119"", ""content"": ""I was getting some random error related to S3 as well 2 days back.\nI tried on VPN then it worked.""}, {""user"": ""aganivada"", ""timestamp"": ""1686150654.422759"", ""content"": ""Hmmm that's weird not aware of any policy that restricts cdk deployment on VPN""}, {""user"": ""aganivada"", ""timestamp"": ""1686150660.712749"", ""content"": ""Let me try through vpn""}, {""user"": ""askumar"", ""timestamp"": ""1686150759.583889"", ""content"": ""Looks like core is not able to start due to bean creation, looks like a new liquibase change is causing conflict.""}, {""user"": ""aganivada"", ""timestamp"": ""1686150816.423799"", ""content"": ""ok you can cancel update stack""}, {""user"": ""aganivada"", ""timestamp"": ""1686150845.813329"", ""content"": ""do we know owner of the changelog?""}, {""user"": ""askumar"", ""timestamp"": ""1686150871.957389"", ""content"": ""checking""}, {""user"": ""askumar"", ""timestamp"": ""1686151059.726149"", ""content"": ""Seems like a discovery change :\n\nliquibase.exception.MigrationFailedException: Migration failed for change set db-patches/changelog/include/db.changelog-1.3.xml::svc-core-data-DISC-3094::\n Reason: liquibase.exception.DatabaseException: ERROR: insert or update on table \""product_instance_user_product_value_team\"" violates foreign key constraint \""fk_user_persona_on_persona\""\n Detail: Key (persona)=(string) is not present in tab\n <@U028EDANJM9> could you please take a look""}, {""user"": ""gshenoy"", ""timestamp"": ""1686151121.082969"", ""content"": ""<@U03KLHDKL1H> this could be due to stale personas in INT. let me check this out""}, {""user"": ""askumar"", ""timestamp"": ""1686151160.911709"", ""content"": ""sure thankyou <@U028EDANJM9> ..let me know if there is any migration related entries missing""}, {""user"": ""aganivada"", ""timestamp"": ""1686151239.403839"", ""content"": ""<@U028EDANJM9> is this related to the query we ran to test in prod yesterday?\n\n```SELECT product_instance_user_id, product_value_team_id, persona, axm_alias\n\tFROM svc_coredata_local_guru.product_instance_user_product_value_team where \n\tpersona not in (select distinct persona_id from svc_coredata_local_guru.persona);```""}, {""user"": ""aganivada"", ""timestamp"": ""1686151313.334399"", ""content"": ""BTW <@U03KLHDKL1H> deployment seems to work after connecting to VPN or may be it is restart will check by disconnecting VPN""}, {""user"": ""gshenoy"", ""timestamp"": ""1686151329.115499"", ""content"": ""It seems like it <@U02BV2DGUKC>, we deployed the hotfix directly in stage so INT was not cleaned up""}, {""user"": ""askumar"", ""timestamp"": ""1686151385.457879"", ""content"": ""yeah <@U02BV2DGUKC> it was weird for me as well...some S3 call was failing and it was failing only for me.. Prashant could deploy it without VPN as well...and we both were in office:sweat_smile:""}, {""user"": ""gshenoy"", ""timestamp"": ""1686151451.048169"", ""content"": ""<@U03KLHDKL1H> it should go through now. There was exactly 1 stale entry""}, {""user"": ""askumar"", ""timestamp"": ""1686151509.685279"", ""content"": ""Thanks <@U028EDANJM9>""}, {""user"": ""askumar"", ""timestamp"": ""1686151763.847399"", ""content"": ""The changelog went through.""}, {""user"": ""aganivada"", ""timestamp"": ""1686199486.237089"", ""content"": "" jfyi I kept running into this issue intermittently deploy works sometimes and sometimes it doesnt and always the root cause was CDK fails when attempting to upload cloud formation to s3. Tried a bunch of things clearing venvs, with/without VPN have to try upgrading mac now based on this long and exhaustive thread :tired_face:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686199583.751589"", ""content"": ""What is your OS version? ""}, {""user"": ""aganivada"", ""timestamp"": ""1686199600.014619"", ""content"": ""11.4""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686199604.235469"", ""content"": ""ok""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686199677.448559"", ""content"": ""I am running 11.6.2""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686199686.972099"", ""content"": ""Never had this issue ""}, {""user"": ""aganivada"", ""timestamp"": ""1686199742.362339"", ""content"": ""This started happening suddenly, I guess may be related to some of the python libraries I had to import for some project""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686199757.639479"", ""content"": ""CDK deployment not working for a platform person is like stethoscope not working for a physician :stuck_out_tongue: ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686199790.250939"", ""content"": ""I started using pyenv""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686199823.817779"", ""content"": ""Give it a try. Helps isolate things nicely ""}, {""user"": ""aganivada"", ""timestamp"": ""1686199839.611289"", ""content"": ""sure yeah this is cool ""}, {""user"": ""aganivada"", ""timestamp"": ""1686199853.773969"", ""content"": ""will give it a try, thank you <@U026PMDB1ND>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1686200067.122149"", ""content"": ""For admin gw I used 3.10 python (don\u2019t remember the reason why, but it was needed for something). With pyenv I just switch to 3.10 in the shell where I start the service and let everything else be at 3.9.""}, {""user"": ""aganivada"", ""timestamp"": ""1686200301.737899"", ""content"": ""ok thats interesting, me and Ashwani ran into this issue but deployment works fine across for everyone and we both have deployed admin-gw recently. I have been deploying all services with 3.9""}]" "1682490267.147919 ","[{""user"": ""aganivada"", ""timestamp"": ""1682490267.147919"", ""content"": ""<@U026PMDB1ND> <@U04JT69T00K> updated deployment dependency template to include i18nexus migration for all future prod deployments cc: <@U02HCMTQU3W>""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1682492533.773329"", ""content"": ""Thanks <@U02BV2DGUKC>""}]" "1680538173.801539 ","[{""user"": ""aganivada"", ""timestamp"": ""1680538173.801539"", ""content"": ""<@U033PPLNFRU> this morning me and <@U0336QZAF98> had a call with Pavan (Haleon) and we asked him to click on invitation again and the flow worked perfectly user was added ads an orgadmin and he didn't see the issue we saw previously. We are not sure what changed from before that auth0 is now accepting the cert, for now we told him that we re added the cert and that might have helped things to work. For now, we can see that the SSO integration is working, Pavan requested to send invitations to following users from his team:\n\n1. - orgadmin \n2. - orgadmin\n3. - orgadmin\nWe could have sent the invites but since the invitation would state it is sent from one of us we thought you can trigger the invitation flow. Please let me know when you have few mins today or tmrw morning IST we can trigger the same flow as we did for Pavan for these 3 users.\n\nPavan also had a couple of suggestions:\n\n1. Teammates is not syncing the updated usernames - Pavan's name in teammates shows with CW extension but his name is now updated without CW so his question was can we sync the user name so they stay updated\n2. show email in collaborators page - currently we are showing collaborators name if name exists otherwise we show collaborators email, Pavan was requesting if we can by default show email column as well or add an option to additionally display emails then it is easy to locate users if users share same name""}, {""user"": ""aganivada"", ""timestamp"": ""1680538438.540259"", ""content"": ""<@U0336QZAF98> metadata backup of users:\n\n\n```{\n \""axm_permissions\"": [\n \""4438::customer::4438:productInstance:13033::readWrite\""\n ]\n}```\n\n```{}```\n\n```{\n\n}```""}, {""user"": ""anair"", ""timestamp"": ""1680564639.145419"", ""content"": ""<@U02BV2DGUKC> ack'ed on the two requests\n1. On the name resyncing, we can probably change just his name for now, on how we globally want to handle it we might need to wait a bit\n2. ack""}]" "1677823170.289689 ","[{""user"": ""pjha"", ""timestamp"": ""1677823170.289689"", ""content"": ""<@U02BV2DGUKC> <@U026PMDB1ND> I am able to figure out the working with lambda function within the VPC . Here I have associated VPC, Subnet & SecurityGroup to the lambda function & made a call to the internal endpoint (endpoint that can be accessed within the VPC ), I got the response back from the api.\nStill working on the documentation once done will post it here.""}, {""user"": ""aganivada"", ""timestamp"": ""1677823237.299639"", ""content"": ""awesome!! thank you <@U04JT69T00K> can you share the lambda link?""}, {""user"": ""pjha"", ""timestamp"": ""1677823295.760169"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1677823354.176519"", ""content"": ""also as discussed please update CDK for one of the existing lambda-functions and move it into vpc. if the MR passes then we can update other lambda functions also.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677823403.806179"", ""content"": ""Good work, <@U04JT69T00K> ""}, {""user"": ""pjha"", ""timestamp"": ""1677823456.386179"", ""content"": ""<@U02BV2DGUKC> sure""}]" "1677623368.389459 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1677623368.389459"", ""content"": "" <@U02BV2DGUKC> - we need to have some way to remember when we turn on debug logs in cloud config. It is impossible to remember, so we need the help of a tool to reset.\n\nWe should also check if we lost the ability to load log4j on the fly. Every time we make a change we are having to bounce (looks like)""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677623534.143789"", ""content"": ""We had a bug popup in PM and since debug was on in prod, we ate up all our quota in < 3 hours. It went into overdrive causing a lot of hits on core""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677623554.625729"", ""content"": ""Though core wasn't in debug, the info logs were significant""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677623599.387319"", ""content"": """"}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677623633.063969"", ""content"": ""The guy gave us some \""overage\"" so that we can start seeing some logs again""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677624018.723489"", ""content"": ""<@U04JT69T00K> - can we set up some precommit hooks for log4j2.xml alone? Have you ever done anything related to automated rollback of a git commit + push after some time?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677624178.060129"", ""content"": ""<Configuration status=\""WARN\"" monitorInterval=\""30\""> should cause the file to be reloaded""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677624227.955779"", ""content"": ""So, we are not pushing the file to the node?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677624241.057619"", ""content"": ""Or is the delay significant?""}, {""user"": ""aganivada"", ""timestamp"": ""1677639990.564249"", ""content"": ""<@U026PMDB1ND>\n\n> we need to have some way to remember when we turn on debug logs in cloud config.\nas a low key tool (temporarily) can we use LS I see that we created an alert for dev env. this should help notify on-call if someone enabled debug on prod.\n\n> We should also check if we lost the ability to load log4j on the fly. Every time we make a change we are having to bounce (looks like)\nit looks like there is something weird about the way log config is loading we tested in stage enabling and disabling access logs couple of days back and it was working without any reload. probably something is missing here. <@U0431DZTPJM> let me know once you are available we can debug this.""}, {""user"": ""aganivada"", ""timestamp"": ""1677640028.311189"", ""content"": ""> So, we are not pushing the file to the node?\n<@U026PMDB1ND> this means the node itself will pull log config every 30 secs""}, {""user"": ""aganivada"", ""timestamp"": ""1677640045.493789"", ""content"": ""checking if there is any issue with cloud-config in delivering log config""}, {""user"": ""ppant"", ""timestamp"": ""1677643899.416759"", ""content"": ""Sure <@U02BV2DGUKC>. Let me know once you are free, we can look into this""}, {""user"": ""aganivada"", ""timestamp"": ""1677771558.989029"", ""content"": ""<@U026PMDB1ND> today me and <@U0431DZTPJM> analyzed the log config for few services it seems like automatic log config changes is working with services that are fully integrated with cloud config (webhook notifications to auto-reload config) to refresh their application.yml changes. So services like tenancy, registry etc are able to pull log config changes without any restart. However, services that are not fully integrated with cloud config are not refreshing log config based on monitor interval and we are having to restart/bounce service to pull the log config changes. It seems like it could be related to a soft reload that spring does whenever a config event is pushed from cloud-config server which is helping services like tenancy to load the log config changes without bounce. I don't recollect verifying log config change refresh for services that were not fully integrated with cloud-config so this issue might have been present even before.\n\nFor now, Prabhu updated core-data also to sync config with cloud-config server and tested locally to make sure log config changes are being pulled automatically by core without refresh. We have to check if integration with cloud-config is required or we are missing anything here. Based on we are setting monitorInterval to 30 secs so service should poll and automatically load log config.\n\nAnother observation <@U0431DZTPJM> made is for tenant level logging we need cloud-config integration + latest observability jar. then we can update log config just mentioning tenant id's where we need debug info without having to enable for all tenants . We verified this also in int today\n\n> Log4j configuration files that specify a monitor interval of greater than zero will use polling to determine whether the configuration has been updated. If the monitor interval is zero then Log4j will listen for notifications from Spring Cloud Config and will check for configuration changes each time an event is generated. If the monitor interval is less than zero Log4j will not check for changes to the logging configuration.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677772202.795579"", ""content"": ""Thanks <@U02BV2DGUKC> and <@U0431DZTPJM> \n\nIs it possible to quickly check if the Collab services are doing the right thing wrt integration with CC? They are generally the ones we tend to tune the most""}, {""user"": ""aganivada"", ""timestamp"": ""1677772663.556489"", ""content"": ""sure <@U026PMDB1ND>""}, {""user"": ""aganivada"", ""timestamp"": ""1677774065.925479"", ""content"": ""CC integration seems to be working for all collab services ""}, {""user"": ""ppant"", ""timestamp"": ""1677822519.266869"", ""content"": ""<@U02BV2DGUKC> I was exploring more on issue for log4j2 not refreshing configs even though `monitorInterval` is set, seems like its a common bug that many people are facing - ""}, {""user"": ""aganivada"", ""timestamp"": ""1677822597.794149"", ""content"": ""Interesting thank you <@U0431DZTPJM>, blocker has been set as not-a-bug :)""}, {""user"": ""ppant"", ""timestamp"": ""1677822786.992579"", ""content"": ""Log4j2 be like \u201cIts not a bug, its a feature\u201d :slightly_smiling_face:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677823349.575979"", ""content"": ""Along with monitorInterval there is a log level we configure. It could be useful to change that to debug in 1-2 services to see if we need more info about how the monitoring is going on. I believe it tends to be very verbose.""}, {""user"": ""aganivada"", ""timestamp"": ""1677823409.875339"", ""content"": ""yeah we can try it locally to see if it has any helpful info""}]" "1676653770.772179 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1676653770.772179"", ""content"": ""Let\u2019s target Monday noon then <@U03DHUAJVMK> and <@U02BV2DGUKC> ""}, {""user"": ""araman"", ""timestamp"": ""1676891795.359889"", ""content"": ""<@U02BV2DGUKC> we can deploy 0.9.7 rel1 once we know the rca for the issue discussed here""}]" "1690059760.859809 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1690059760.859809"", ""content"": "" Our RDS spend in INT is much better now compared to a few months back (in part, but not all all due to the move to Aurora V2)""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690060160.820709"", ""content"": "" looks like something has changed in the last few days because of which we are using more RDS than in the first three days of the week.\n\nThere was a spike in estimation queries (fyi ) on the 18th, but the sustained increase is on the PM side.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690060280.112419"", ""content"": ""7/18 6:00 to 7/22 06:00""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690060357.275359"", ""content"": ""7/16 00:00 to 7/18 06:00""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690060370.713689"", ""content"": ""Very different load profiles""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690060783.470059"", ""content"": ""<@U026PMDB1ND> - can you share the link to aws console - i am interested in estimation service sql query - which api path is triggering it""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690060867.710909"", ""content"": ""<@U02BV2DGUKC> - maybe we should have a task in the plat team to measure perf insights seen as $ cost of the key elements week over week (compare current week to previous week and also to 4 weeks back) . If we can also use non-$ metrics, that's fine too, but $ comparison would be the base requirement.\n\nWe could run this every Sunday at 6 am PST, store the current values in dynamo and send out a report to a slack channel with the simple trend information.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690060879.142229"", ""content"": ""<@U0336QZAF98> - \n\nINT account""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690060923.607409"", ""content"": ""<@U0336QZAF98> - since it was a one-off I am assuming it was an experiment we ran on that day""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1690060980.786339"", ""content"": ""sure <@U026PMDB1ND>.will keep a watch on this""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690065527.313269"", ""content"": ""> maybe we should have a task in the plat team to measure perf insights seen as $ cost of the key elements week over week (compare current week to previous week and also to 4 weeks back)\nLooks like we just need to configure this: and run a GL job every week to read the last week's data and the data, previous week's data from data from 4 weeks back and just focus on key changes and report on that.\n\nBasically, we shouldn't report on every item, but just the top few that have the biggest deltas ($ and %).""}, {""user"": ""aganivada"", ""timestamp"": ""1690172362.791189"", ""content"": ""> maybe we should have a task in the plat team to measure perf insights seen as $ cost of the key elements week over week\nsure <@U026PMDB1ND> , will add a task cc: <@U04JT69T00K>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690255970.217879"", ""content"": ""<@U04JT69T00K> - if you setup CUR into a private bucket - need to see if we need to do this once per env or if there's a way to do it for multiple accounts from one place (we could open a case with AWS to get some clarity <@U02BV2DGUKC>) and check that it works as expected, we can figure out how to parse, store the results and generate comparison reports.""}]" "1690523608.265049 ","[{""user"": ""aganivada"", ""timestamp"": ""1690523608.265049"", ""content"": ""<@U026PMDB1ND> <@U04JT69T00K> P81 by default seems to logout users automatically after 30 days updating this to 60 days which is the maximum allowed time""}, {""user"": ""aganivada"", ""timestamp"": ""1690523618.883449"", ""content"": """"}]" "1691080034.249959 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1691080034.249959"", ""content"": ""<@U026PMDB1ND> - celery supports horizontal as well as vertical scaling\nA Celery system can consist of multiple workers and brokers, giving way to high availability and horizontal scaling.\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1691084295.537429"", ""content"": ""<@U0336QZAF98> - thanks. I think we should figure out how this would apply to our setup (ie. would we auto-scale or scale up manually? What would that process look like and more fundamentally, is there any issue with horizontal scaling combined with the use of SQS)""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1691084573.351179"", ""content"": ""We went over at high level for NLP design given by <@U040RCBPBEC> - We planned for episode 2) but it did not happen and ciso started - right now i have configured autoscaling based on length of pending msg in sqs queue\n``` qps_scaling_steps = make_static_param(fixed_value=[\n {\""lower\"": 1, \""change\"": 0}, # 1 msg = 1 worker (default 1 worker)\n {\""lower\"": 50, \""change\"": +1}, # 50 msgs = 2 workers\n {\""lower\"": 100, \""change\"": +1}, # 100 msgs = 3 workers\n {\""lower\"": 150, \""change\"": +2}, # 150 msgs = 5 workers\n ])```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1691084618.911289"", ""content"": ""<@U0336QZAF98> - did you see this kicking in?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1691084619.196379"", ""content"": ""i completely forgot i did this in cdk..my bad""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1691084644.367689"", ""content"": ""No i am yet to test this \u2026will plan for it""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1691084774.105189"", ""content"": ""I guess we need to see this kick in and work okay, otherwise we could get into weird situations. I suppose the config above would ensure that we don't go beyond 5 (so no chance of a runaway). Would like to see if the scale down happens properly.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1691084906.881949"", ""content"": ""Agreed <@U026PMDB1ND>""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1691167420.056369"", ""content"": ""<@U026PMDB1ND> - i have verified scaling - i needed to do couple of fixes - Below is MR for the same - i have documented the same in \n""}]" "1673478411.493089 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1673478411.493089"", ""content"": "" - do we know if the Job executor service is running as it is expected to and is succeeding? I am not sure if we are tracking the jobs it is supposed to run (some cleanup, archival etc.)""}, {""user"": ""aganivada"", ""timestamp"": ""1673581646.917489"", ""content"": ""<@U03KLHDKL1H> /<@U0431DZTPJM> do you want to take a stab at reviewing health of job executors ?""}, {""user"": ""askumar"", ""timestamp"": ""1673583097.939619"", ""content"": ""Sure <@U02BV2DGUKC>""}, {""user"": ""ppant"", ""timestamp"": ""1673583394.000009"", ""content"": ""Sure""}, {""user"": ""aganivada"", ""timestamp"": ""1673583514.597539"", ""content"": ""Prabhu since Ashwani is currently on on-call (and placed early bid :slightly_smiling_face: ) we can may be have Ashwani look into it""}, {""user"": ""ppant"", ""timestamp"": ""1673583572.875949"", ""content"": ""Haha sure. No issues :smile:""}, {""user"": ""aganivada"", ""timestamp"": ""1673583595.566259"", ""content"": ""<@U03KLHDKL1H> this is the project ""}]" "1687245312.949629 ","[{""user"": ""ppant"", ""timestamp"": ""1687245312.949629"", ""content"": "" Please review this document containing the structure of multi instance config in SSM cc: <@U04JT69T00K>""}, {""user"": ""ppant"", ""timestamp"": ""1687408955.299509"", ""content"": "" Just a reminder""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687479577.029949"", ""content"": ""<@U0431DZTPJM> - looks good. Thanks for thinking this through.\n\n Can we add the name of the $USER who is uploading the config and the time ($(date)) of upload also in the JSON. For example:\n\n```{\n \""tenancy_service\"": \""plat-4\"",\n \""onboarding_app_service\"": \""plat-2\""\n \""__user\"": \""ppant\"",\n \""__date\"": \""Thu Jun 22 17:15:23 PDT 2023\""\n}```\nAlso add an option to read and print the parameter? For example, if we invoke it like this `./ssm-config.sh int axmint 19190` we should treat it as a read operation.\n\nFinally, add this script in cdk-artifacts repo in apps/scripts directory.""}, {""user"": ""ppant"", ""timestamp"": ""1687494497.044679"", ""content"": ""Sure <@U026PMDB1ND>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687494682.330739"", ""content"": ""<@U0431DZTPJM> if no tenant is given maybe we can get all params in that path. Could be useful too.""}, {""user"": ""ppant"", ""timestamp"": ""1687494845.475239"", ""content"": ""<@U026PMDB1ND> By this do you mean getting all the SSM parameters for multi instance? Like if there are only two tenants T1 and T2 in SSM, then fetch `/int/multi-instance-config/T1` and `/int/multi-instance-config/T2` ?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687494990.696269"", ""content"": ""Yes. Do you think it makes sense to do it that way? We could make a json like below and print it:\n\n{\nt1: {\u2026.},\nt2: {\u2026..}\n}""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687495038.854069"", ""content"": ""Allows us to pipe to jq if needsd""}, {""user"": ""ppant"", ""timestamp"": ""1687495140.960959"", ""content"": ""Sure, we can do this. Just have to see if we can fetch SSM parameters using regex like `/int/mulit-instance-config/**`""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687496550.243169"", ""content"": ""Yeah.. or have to brute force it (with paging). It is not likely for this mode to be used often times so it might be okay even if it is a bit slow""}]" "1679383614.060059 ","[{""user"": ""pjha"", ""timestamp"": ""1679383614.060059"", ""content"": ""Hey, <@U026PMDB1ND> here one backend-service calls the other through private endpoint, which routes to the ECS Tasks since Route53 values is ECS task ip address I was wondering why we don't have LoadBalancer in place of IP address in the route53.\n In order to approach multi instance deployment , when one backend-service calls the other we wanted have condition(rules) where it can routes to different version, which looks possible though LoadBalancer(based on http headers).\n Could you please let me know the rational behind having ip addresses in place of loadbalancer.\ncc: <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1679383941.643089"", ""content"": ""We compared the number of requests made to LB vs requests in the access log of tenancy. Tenancy is receiving a much higher number of requests compared to what we see in LB it seems requests are not going through the load balancer, route53 is directly calling the ecs task on the private domain.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679411847.568869"", ""content"": ""Are they health requests, <@U02BV2DGUKC> /""}, {""user"": ""aganivada"", ""timestamp"": ""1679412295.841139"", ""content"": ""<@U026PMDB1ND> I dont think LB is showing health requests in monitoring most likely the requests that are made directly from api-gw are ending up at LB while regular backend to backend are reaching service directly\n\nLB => \n\nlogz => ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679413001.339459"", ""content"": ""We must be using the DNS load balancing for within VPC access. We can probably move to going via the LB.""}, {""user"": ""aganivada"", ""timestamp"": ""1679413192.034939"", ""content"": ""would it make sense to test with a couple of services manually before updating CDK?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679414088.835589"", ""content"": ""We'd need to register the LB with the service name - probably use something different from what we are doing now (say tenancy-lb vs tenancy) and update CDK to give out pointers to tenancy-lb going forward. That way we can coexist""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679414555.304219"", ""content"": ""We can deploy all the services once when we are ready to switch from xxx to xxx-lb across the board.""}, {""user"": ""aganivada"", ""timestamp"": ""1679414686.419779"", ""content"": ""> We'd need to register the LB with the service name\nin cloud-map <@U026PMDB1ND>?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679414801.925729"", ""content"": ""yeah""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679414944.410179"", ""content"": ""I think that's how it would work but we need to see if there's any other recommended way""}, {""user"": ""aganivada"", ""timestamp"": ""1679415011.384249"", ""content"": ""ok makes sense, found this article \n\n> AWS_ALIAS_DNS_NAME\n> If you want AWS Cloud Map to create a Route 53 alias record that routes traffic to an Elastic Load Balancing load balancer, specify the DNS name that's associated with the load balancer. For information about how to get the DNS name, see DNSName> in the _Route 53 API Reference_.\n> Note the following:\n> \u2022 The configuration for the service that's specified by `ServiceId` must include settings for an `A` record, an `AAAA` record, or both.\n> \u2022 In the service that's specified by `ServiceId`, the value of `RoutingPolicy` must be `WEIGHTED`.\n> \u2022 If the service that's specified by `ServiceId` includes `HealthCheckConfig` settings, AWS Cloud Map creates the health check, but it won't associate the health check with the alias record.\n> \u2022 Auto naming currently doesn't support creating alias records that route traffic to AWS resources other than ELB load balancers.\n> \u2022 If you specify a value for `AWS_ALIAS_DNS_NAME`, don't specify values for any of the `AWS_INSTANCE` attributes.\n> \u2022 \n> ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679415085.409429"", ""content"": ""Yeah.. I think we should look at the CDk v1 reference for doing the same, make sure we have a path and then try it out manually""}, {""user"": ""aganivada"", ""timestamp"": ""1679415252.192259"", ""content"": """"}]" "1689315398.223649 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1689315398.223649"", ""content"": "" - injury in my left hand - cartilage tear- will connect from home till recovery.""}, {""user"": ""akasim"", ""timestamp"": ""1689315694.687239"", ""content"": ""Take care <@U0336QZAF98>\nbadminton?""}, {""user"": ""aganivada"", ""timestamp"": ""1689316047.231979"", ""content"": ""Take care <@U0336QZAF98>""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1689316064.075889"", ""content"": ""Bike accident :disappointed:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1689317035.394259"", ""content"": ""Get well soon, <@U0336QZAF98> ""}, {""user"": ""askumar"", ""timestamp"": ""1689317672.651579"", ""content"": ""Take care <@U0336QZAF98>""}, {""user"": ""hsingh"", ""timestamp"": ""1689320886.555019"", ""content"": ""Take care""}]" "1681317539.108249 ","[{""user"": ""askumar"", ""timestamp"": ""1681317539.108249"", ""content"": "" <@U026PMDB1ND> <@U02BV2DGUKC>\nInitiating thread here to decide on the approach for Data enrichment of snowflake data :\nEg : TenantId to Name map, UserId to Name map\n\nApproach 1 :\nLambda enriching the event by making call to backend for every event it receives before pushing data.\n\nApproach 2 :\nSnowflake having table with these key value maps.\nHere again we have two sub approaches\n a. A different Lambda pushing this data via a job to snowflake.\n b. We take a dump from DB and upload file either manually or via script again.\n\nRelated discussion\n\n\nPlease add your views/comments""}, {""user"": ""aganivada"", ""timestamp"": ""1681359226.579489"", ""content"": ""<@U03KLHDKL1H> for 2 do we need a different lambda? can we use the same lambda that pushes other events to also push tenant creation and user creation events?""}, {""user"": ""askumar"", ""timestamp"": ""1681359666.825409"", ""content"": ""Yes...we can definitely use same lambda <@U02BV2DGUKC> if it comes in form of event.""}]" "1676029528.861739 ","[{""user"": ""rvaidya"", ""timestamp"": ""1676029528.861739"", ""content"": ""<@U02BV2DGUKC> seeing some weird issue with one of the app calls in STAGE (trace-id: 63e61cc5f56c621a380900d2dee4c6de): /tenant/user/info\nThe App is calling multiple downstream API for user: with tenant: 15328 \u2026 but somehwre core is using user : with tenant : 16992 on the same trace-id. One of the API call is failing with permission issue cos alexw doesnt have permissions on tenant 15238.\nwe can huddle to explain the issue better.\n<@U02HCMTQU3W> <@U0336QZAF98> <@U03DHUAJVMK> <@U03DLS1FN3D>""}, {""user"": ""aganivada"", ""timestamp"": ""1676034110.957979"", ""content"": ""sure <@U02GC8SE18V>, checking the trace-id meanwhile""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1676034141.282869"", ""content"": ""<@U02BV2DGUKC> - let me know if u r available - we can discuss""}, {""user"": ""aganivada"", ""timestamp"": ""1676034155.425199"", ""content"": ""we can discuss ~not~ now*Seshan""}, {""user"": ""rvaidya"", ""timestamp"": ""1676034180.960919"", ""content"": ""loop me in as well""}, {""user"": ""aganivada"", ""timestamp"": ""1676036559.888049"", ""content"": ""<@U02GC8SE18V> is this intermittent or happening continously?""}, {""user"": ""aganivada"", ""timestamp"": ""1676036572.617659"", ""content"": """"}, {""user"": ""rvaidya"", ""timestamp"": ""1676037610.220349"", ""content"": ""<@U02BV2DGUKC> it is intermittent\u2026but as <@U0336QZAF98> showed, it happened few times in last few days""}, {""user"": ""aganivada"", ""timestamp"": ""1676038286.385849"", ""content"": ""confirmed that we are clearing data from thread context during post-handle in UserContextInterceptor\n\n> ```@Override\n> public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler,\n> ModelAndView modelAndView) throws Exception {\n> log.debug(\""post-handle invoked, resetting attributes from request\"");\n> RequestContextHolder.resetRequestAttributes();\n> ThreadContext.remove(UserContextConstants.AXM_USER_ID);\n> ThreadContext.remove(UserContextConstants.AXM_TENANT_ID);\n> ThreadContext.remove(UserContextConstants.AXM_USER);\n> }```\n> \nalso confirmed that same thread is having different users so user not most likely not getting cached at thread level \n\nchecking for other possibilities""}, {""user"": ""aganivada"", ""timestamp"": ""1676041148.573929"", ""content"": ""<@U0431DZTPJM> we want to enable debug loggin for tenant 16992 in stage do we have the steps logged ?""}, {""user"": ""ppant"", ""timestamp"": ""1676041355.784359"", ""content"": ""Sure Anil, the confluence doc is here . You just need to add the filter and log level to debug""}, {""user"": ""aganivada"", ""timestamp"": ""1676041651.000689"", ""content"": ""thank you <@U0431DZTPJM>, is this also required?\n\n> packages=\""com.axm.platform.logging.util\"">""}, {""user"": ""ppant"", ""timestamp"": ""1676041748.842699"", ""content"": ""Yup""}, {""user"": ""aganivada"", ""timestamp"": ""1676041809.729949"", ""content"": ""cool, <@U0431DZTPJM> can you verify this config? ""}, {""user"": ""ppant"", ""timestamp"": ""1676042002.326209"", ""content"": ""LGTM :+1:""}, {""user"": ""aganivada"", ""timestamp"": ""1676042154.351499"", ""content"": ""is bounce required?""}, {""user"": ""ppant"", ""timestamp"": ""1676042525.059699"", ""content"": ""Nope but if it does not work, please try with bounce. I only tried refresh scope locally""}, {""user"": ""aganivada"", ""timestamp"": ""1676042550.018229"", ""content"": ""hmmm I dont think core has refresh scope enabled""}, {""user"": ""aganivada"", ""timestamp"": ""1676042575.043779"", ""content"": ""but this should be auto refresh for logging""}, {""user"": ""rvaidya"", ""timestamp"": ""1676257464.054599"", ""content"": ""<@U02BV2DGUKC> <@U0336QZAF98> <@U0431DZTPJM> any further finding on this?""}, {""user"": ""rvaidya"", ""timestamp"": ""1676257538.374699"", ""content"": ""Looks like <@U040RCBPBEC> also faced similar issue : ""}, {""user"": ""aganivada"", ""timestamp"": ""1676259247.263749"", ""content"": ""Yes <@U02GC8SE18V> looks like this is happening when threads are getting stuck due to failed users (wrong or empty permissions) and there are additional requests being sent to same thread. I am trying to get some logs to validate this""}, {""user"": ""aganivada"", ""timestamp"": ""1676273727.647889"", ""content"": ""<@U02GC8SE18V>, <@U0336QZAF98> <@U0431DZTPJM> found the rootcause, turns out the issue is we are using posthandle to cleanup MDC and reset request context attributes in UserContextInterceptor in axm-commons but posthandle does not get invoked when there is a error during execution. So in this case when discovery sets everything up during pre-handle and invokes core to make a request with a faulty user, core returns 500 and spring treats execution failed so it retries 3 times and doesn't invoke post-handle method which is supposed to cleanup the request context, this results in some random call in future from discovery to core failing because discovery thread will attempt to pass the faulty token set in MDC to core.\n\nInterestingly after re-creating the scenario in int and enabling logs I was not able to reproduce the issue since int has a fix where accessdenied exception is 403 instead of 500 which seems to be considered as a valid response and discovery is not retrying it 3 times the way it happens in stage.\n\nFix for the issue is to use afterCompletion() rather than postHandle() to cleanup the context in MDC\n\n\n\n> afterCompletion => Callback after completion of request processing, that is, after rendering the view. Will be called on any outcome of handler execution, thus allows for proper resource cleanup.\n> \n> postHandle => Interception point after successful execution of a handler. Called after HandlerAdapter actually invoked the handler, but before the DispatcherServlet renders the view. Can expose additional model objects to the view via the given ModelAndView.\nso when can this happen? this can happen if discovery calls core and core returns 500 (due to permission or any other issue) and the context is not getting cleared. we never had such situation before because we didn't have product admin's with empty permissions until we did collaborators testing in stage with empty permissions for Alexw user (this is fixed now).\n\nI will be adding an alert for this but based on logz so far there are no such errors in prod (or int) most likely because we didnt create users of role product admin with empty permissions.\n\n\n\nwill be publishing a HF with the commons change and deploy discovery, core and tenancy as part of 0.9.7-rel1\n\n<@U03KLHDKL1H> can we also add the fix for response code 403 (instead of 500) when there is an access issue to 0.9.7-rel1?\n\ncc: <@U03DHUAJVMK> <@U040RCBPBEC>""}, {""user"": ""askumar"", ""timestamp"": ""1676274207.065449"", ""content"": ""Thankyou <@U02BV2DGUKC> this has been really weird and tough to figure out.\nWill add the changes for 403 to core""}, {""user"": ""rvaidya"", ""timestamp"": ""1676275289.261279"", ""content"": ""<@U02BV2DGUKC> One question which is confusing for me :\nEven if the MDC is not clearing the old tenant/user, why are we using the tenant/user from MDC context for querying the db? Shouldn\u2019t the quering data use tenat/user from the request body/params?\ncc <@U02HCMTQU3W>""}, {""user"": ""aganivada"", ""timestamp"": ""1676276247.791039"", ""content"": ""> Even if the MDC is not clearing the old tenant/user, why are we using the tenant/user from MDC context for querying the db? Shouldn\u2019t the quering data use tenat/user from the request body/params?\n<@U02GC8SE18V> we get user and permissions from x-axm-id-token header token only for checking whether user is allowed to read product instance and in this case we are getting a proper request but wrong header forcing us to think this might be a forged request. For backend services axm-id-token header is the only way to figure out the user and if that is forged/cached due to MDC or any other issue backend will have no way to figure out the original caller.""}, {""user"": ""askumar"", ""timestamp"": ""1676279877.535099"", ""content"": ""<@U02BV2DGUKC> I have added changes for Core with HF version \n\nYou may please deploy it to stage and we can test it out.""}, {""user"": ""aganivada"", ""timestamp"": ""1676281045.013389"", ""content"": ""sure <@U03KLHDKL1H>""}, {""user"": ""svummidi"", ""timestamp"": ""1676312364.736109"", ""content"": ""<@U02BV2DGUKC> - I have similar question as <@U02GC8SE18V>, why details from the request header not taking precedence over MDC? Are we clearing this MDC at the beginning of the Http thread also? We need to cover all the cases of using stale user context from previous request, it will create serious security issues.""}, {""user"": ""aganivada"", ""timestamp"": ""1676347167.140789"", ""content"": ""<@U040RCBPBEC>\n\n> why details from the request header not taking precedence over MDC?\nsorry I got a bit confused with terminology though we do store context in MDC when we call another service we are using requestcontextholder. when a backend receives a request we store the axm-id token sent from apps (actual user token) in requestcontextholder and use the data in this header to process request all of this happens in UserContextInterceptor in axm-commons.\n\n> ```RequestContextHolder.setRequestAttributes(requestScopeAttr, true);```\n\nThis header is separate from the usual S2S token, now when backend needs to call another backend service since that other service also might need user context so we forward the header value initially stored in requestcontextholder this happens in webclientconfig in axm-commons\n\n> ```return WebClient.builder().filter(oauth)\n> .filter(ExchangeFilterFunction.ofRequestProcessor(\n> request -> {\n> log.debug(\""Headers are:{}\"", request.headers().keySet());\n> if (RequestContextHolder.getRequestAttributes() != null) {\n> var token = RequestContextHolder.getRequestAttributes()\n> .getAttribute(UserContextConstants.X_CUSTOM_TOKEN_HEADER,\n> RequestAttributes.SCOPE_REQUEST);```\n> \nthis is how we are forwarding user context from one service to another when there are chain of services invoved in addressing a user request. The failures we observed here are authorization issues because somewhere down the line wrong authorization is being forwarded to next service.\n\n> Are we clearing this MDC at the beginning of the Http thread also?\nWe are using HttpServletRequest.setAttribute and getAttribute to access the decoded authorization info. this info gets populated by interceptor in pre-handle and should have been cleared by post-handle but we didnt realize that post-handle doesn't invoke in case of API failures now we are fixing this with afterCompletion.\n\n> We need to cover all the cases of using stale user context from previous request, it will create serious security issues.\nagreed, me and Anu discussed about this yesterday we can replicate some cases by manually introducing failures. However, we might also need to find some other negative cases to test this end-to-end""}]" "1681435534.634889 ","[{""user"": ""ppant"", ""timestamp"": ""1681435534.634889"", ""content"": ""<@U026PMDB1ND> During synth/deploy of a CDK apps stack, why are they dependent on these parameter stores from SSM, particularly the third one _`sqs-lambda-listener`_? This is from synth of tenancy\n```context: {\n 'ssm:account=433798924509:parameterName=/int/hosted-zone-id:region=us-west-2': 'Z00551842EESEDBMCLRWZ',\n 'ssm:account=433798924509:parameterName=/int/axm-vpc:region=us-west-2': 'vpc-0be4125b86969d836',\n 'ssm:account=433798924509:parameterName=/int/sqs-lambda-listener:region=us-west-2': 'arn:aws:lambda:us-west-2:433798924509:function:message-broker-sqs-lambda',\n 'ssm:account=433798924509:parameterName=/int/secret-for-launchdarkly-api-key-arn:region=us-west-2': 'arn:aws:secretsmanager:us-west-2:433798924509:secret:/int/secret-for-launchdarkly-api-key-ha21hh',\n 'ssm:account=433798924509:parameterName=/int/dashboard-url:region=us-west-2': '',\n 'ssm:account=433798924509:parameterName=/int/client-secret-for-auth0-arn:region=us-west-2': 'arn:aws:secretsmanager:us-west-2:433798924509:secret:/int/client-secret-for-auth0-MN7zNV',\n 'ssm:account=433798924509:parameterName=/int/otel-collector-config:region=us-west-2': 'arn:aws:secretsmanager:us-west-2:433798924509:secret:/int/otel-collector-config-Z36jAM',\n 'ssm:account=433798924509:parameterName=/int/secret-for-lightstep-token-arn:region=us-west-2': 'arn:aws:secretsmanager:us-west-2:433798924509:secret:/int/secret-for-lightstep-token-eRmRTl',\n 'ssm:account=433798924509:parameterName=/int/secret-for-grafana-api-key-arn:region=us-west-2': 'arn:aws:secretsmanager:us-west-2:433798924509:secret:/int/secret-for-grafana-api-key-9dBhia',\n 'ssm:account=433798924509:parameterName=/int/prometheus-server:region=us-west-2': '',\n 'ssm:account=433798924509:parameterName=/int/prometheus-username:region=us-west-2': '432668',\n 'ssm:account=433798924509:parameterName=/int/vpc-link-id:region=us-west-2': 'kt5idy',\n 'ssm:account=433798924509:parameterName=/int/http-api-id:region=us-west-2': 'lsb5xc30t2',\n}```""}, {""user"": ""aganivada"", ""timestamp"": ""1681446913.702199"", ""content"": ""<@U0431DZTPJM> most of these props are common, regarding sqs-lambda I think we may have reused some core library references of these common core stacks so cdk is injecting the property. Surely tenancy doesn't use \""message-broker-sqs-lambda\""""}, {""user"": ""ppant"", ""timestamp"": ""1681448222.046549"", ""content"": ""Currently message-broker-sqs-lambda SSM param is applied to all apps stack regardless. It is happening because whenever we build a `BaseAppStack` and call its `execute` function, inside this `apply_sqs_lambda_defaults_from_context` method is invoked. What this method does is it populates the SSM param for message-broker-sqs-lambda in all the apps stack whenever execute is called (as core stacks are kind of like foundation and, I guess, are built before apps stack so this SSM param is always present). So yesterday when we destroyed the sqs lambda from core, synth/deploy started to fail for all apps stack as this SSM param was deleted.""}, {""user"": ""ppant"", ""timestamp"": ""1681448261.048709"", ""content"": ""Do we know if there are any services that directly depend on this?""}, {""user"": ""aganivada"", ""timestamp"": ""1681449174.065569"", ""content"": ""<@U0431DZTPJM> we have to check on this with not sure about why would a service depend on lambda stack or may be it was just a mis-config""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1681452876.658099"", ""content"": ""<@U0431DZTPJM> only message-broker depends on message-broker-sqs-lambda""}]" "1679412406.904519 ","[{""user"": ""pjha"", ""timestamp"": ""1679412406.904519"", ""content"": "" Here is the finding Documentation for the 'AutoDeploying Docker image to the ECR' ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679415313.868229"", ""content"": ""Let's take this to a logical conclusion to see what would that lambda look like, what are the permissions required and so on.""}]" "1677348176.055279 ","[{""user"": ""araman"", ""timestamp"": ""1677348176.055279"", ""content"": ""<@U02BV2DGUKC> <@U026PMDB1ND> Facing 401 (auth0 exception) while navigating to Actions page on stage in a tenant that was working fine until 0.9.8. ""}, {""user"": ""aganivada"", ""timestamp"": ""1677380555.515399"", ""content"": ""<@U03DHUAJVMK> can we check if there is any issue with organization? Does general dashboard work for this tenant?""}, {""user"": ""aganivada"", ""timestamp"": ""1677380649.680399"", ""content"": ""Please share creds will take a look""}, {""user"": ""bganganna"", ""timestamp"": ""1677385121.464809"", ""content"": ""<@U02BV2DGUKC> yday even I saw this issue while loading vendor dashboard, post reload it worked fine .. 63fa242fab6904b3d0389e582d3b04dd and 63fa245f051056e92b9c57525b16bb30 are traceIDS during that time. cc <@U02GC8SE18V>""}, {""user"": ""bganganna"", ""timestamp"": ""1677385316.521969"", ""content"": """"}, {""user"": ""araman"", ""timestamp"": ""1677397303.473659"", ""content"": ""<@U02BV2DGUKC> shared creds in dm""}, {""user"": ""aganivada"", ""timestamp"": ""1677410605.756269"", ""content"": ""<@U03NZ7Z52S2> /<@U03DHUAJVMK> initial observation is that tenancy got requests to fetch organization details of orgs that don't exist or have been deleted from auth0\n\nhere is a logz search for all such failures in stage \n\nand based on DB these tenants have been deleted. is it possible that these users were vendors of some customer orgs which were later deleted?""}, {""user"": ""araman"", ""timestamp"": ""1677418181.873419"", ""content"": ""the tenant in which I faced the issue is 17875 and am able to access dashboard fine""}, {""user"": ""araman"", ""timestamp"": ""1677418264.903189"", ""content"": ""and action creation was working fine until 0.9.8""}, {""user"": ""aganivada"", ""timestamp"": ""1677418450.091749"", ""content"": ""<@U03DHUAJVMK> while accessing useractions it seems we are checking connection between vendor and customer and it seems there were some old connections with deleted tenants that seem to be creating issue. Not sure if something changes in useractions flow in 0.9.8 will try to find reference, atleast from logs i can see in all failed attempts the tenant that was being queried was deleted, it might not be the tenant user is logging in to but some old actions or something that this tenant was associated with might be creating the issue.""}, {""user"": ""araman"", ""timestamp"": ""1677418555.999789"", ""content"": ""vendor tenants may be deleted. User actions (stale) might be there. so fetching them is throwing this error?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677442847.674179"", ""content"": ""<@U02HQ78V9A5> what is the expected behavior in this situation ""}, {""user"": ""bganganna"", ""timestamp"": ""1677469321.698629"", ""content"": ""<@U02BV2DGUKC> For my tenant , product page failed and post reloading it worked fine.""}, {""user"": ""aganivada"", ""timestamp"": ""1677470235.465059"", ""content"": ""hmmm quick call <@U03NZ7Z52S2>? I thought I saw similar exception""}, {""user"": ""bganganna"", ""timestamp"": ""1677470446.931799"", ""content"": ""sure <@U02BV2DGUKC>""}, {""user"": ""rvaidya"", ""timestamp"": ""1677470623.803619"", ""content"": ""lmk if u need me in the call ?""}, {""user"": ""aganivada"", ""timestamp"": ""1677470697.037949"", ""content"": ""<@U02GC8SE18V> for the issue Bhavana noticed we think it is mostly due to db unavailability. will need your inputs on the user actions issue""}, {""user"": ""rvaidya"", ""timestamp"": ""1677472404.983279"", ""content"": ""<@U03DHUAJVMK>, me and <@U02BV2DGUKC> discussed this : I think there can be 2 different issues here :\n1. Handle tenant deletion gracefully in Apps and not fail like this.\n2. Handle tenant deletion in UAS and see why it wasnt handled with some references still present in MB/UAS. Was this tenant deleted using the teantn deletion framework? Also is it a usecase to handle? cc <@U02HCMTQU3W> <@U03BPNY5AGM> <@U02HQ78V9A5> ""}, {""user"": ""aganivada"", ""timestamp"": ""1677472710.793569"", ""content"": ""deleted tenants that are creating issues are 16993,17137, 17139 though the tenants are deleted MB mapping still exists, this is probably the reason why we are attempting to fetch info from tenancy and failing since the org is already deleted\n\n> SELECT id, pvt_id, vvt_id, vendor_tenant_id, customer_tenant_id\n> \tFROM svc_message_broker_default_axm.customer_vendor_team_map \n> \twhere vendor_tenant_id in (16993,17137, 17139 ) or \n> \tcustomer_tenant_id in (16993, 17137, 17139);\n""}, {""user"": ""akasim"", ""timestamp"": ""1677472730.140449"", ""content"": ""<@U02BV2DGUKC> <@U03NZ7Z52S2> pls add me in the call if its happening""}, {""user"": ""aganivada"", ""timestamp"": ""1677472765.669159"", ""content"": ""let me call you <@U02HQ78V9A5>""}, {""user"": ""aganivada"", ""timestamp"": ""1677472785.189379"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1677473948.394669"", ""content"": ""<@U03DHUAJVMK> for (2) these tenants 16993,17137, 17139 seem to have been removed only from tenancy but no other backend service (core, MB etc) we can still see PVT's for these tenants. based on tenancy it seems these tenants were deleted a while ago can you recollect if there was any issue deleting these tenants like FF not being enabled? unfortunately logz only keeps track of last 15 days logs so we dont have details when these tenants were deleted cc: <@U02GC8SE18V> <@U02HQ78V9A5>""}, {""user"": ""araman"", ""timestamp"": ""1677474522.937609"", ""content"": ""I will connect with you over a cll <@U02BV2DGUKC> to better understand this.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677517593.087589"", ""content"": ""<@U02BV2DGUKC> - what was the final analysis? Could this happen to any of our prod customers?""}, {""user"": ""aganivada"", ""timestamp"": ""1677518104.751439"", ""content"": ""<@U026PMDB1ND> this seems to have caused since we are displaying tenant name in user actions (recent change) in 0.9.8 from apps, earlier we were showing product name which would work even if there is stale tenant data.\n\nWe decided to fix it as a bug in 0.9.8 from apps to handle deleted tenants in connected-products flow (1) from <@U02GC8SE18V>'s message above, for now to unblock <@U03DHUAJVMK> we have re-trigerred tenant deletion where cleanup did not happen properly earlier. If we fix this in 0.9.8 we should probably not see this in prod customers.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677519368.413029"", ""content"": ""Thanks <@U02BV2DGUKC>""}, {""user"": ""araman"", ""timestamp"": ""1679708533.277049"", ""content"": ""<@U02BV2DGUKC> <@U02GC8SE18V> was this fixed in 0.9.8 or 0.9.9?""}, {""user"": ""rvaidya"", ""timestamp"": ""1679712491.318389"", ""content"": ""9.9 <@U03DHUAJVMK>""}]" "1676565297.122469 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1676565297.122469"", ""content"": ""<@U02BV2DGUKC> - what is the fastest way to get a tenant id given a name?""}, {""user"": ""rvaidya"", ""timestamp"": ""1676565433.309299"", ""content"": "" with ilike rsql queryfilter ?""}, {""user"": ""aganivada"", ""timestamp"": ""1676565519.821839"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1676565524.443959"", ""content"": ""yup it does support rsql""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1676567557.708189"", ""content"": ""<@U02BV2DGUKC> <@U02GC8SE18V> ""}]" "1684241669.302749 ","[{""user"": ""aganivada"", ""timestamp"": ""1684241669.302749"", ""content"": ""<@U03KLHDKL1H> where are we with int db migration?""}, {""user"": ""askumar"", ""timestamp"": ""1684241939.480849"", ""content"": ""Hey <@U02BV2DGUKC> \nThere is one change in CDK that I am doing for using the same secret Id beacuse we are not creating new param in secret manager.\nThis will be done by tomorrow.\nFor changing the password <@U04JT69T00K> is looking into it.\nI am finishing up the migration plan document today.\nSo we should be good to go from tomorrow, since we have branch cut as well for the release.""}, {""user"": ""aganivada"", ""timestamp"": ""1684243632.179479"", ""content"": ""ok thank you <@U03KLHDKL1H>, will we be done with migration before branch-cut?""}, {""user"": ""askumar"", ""timestamp"": ""1684243690.171389"", ""content"": ""yes <@U02BV2DGUKC> for Core it will be done""}, {""user"": ""aganivada"", ""timestamp"": ""1684243732.241729"", ""content"": ""ok lets have a plan in place for core in stage also""}, {""user"": ""askumar"", ""timestamp"": ""1684243773.501779"", ""content"": ""sure <@U02BV2DGUKC>, since the steps will be on similar lines, so I will add additional steps if any for stage as well :slightly_smiling_face:""}]" "1691669913.833899 ","[{""user"": ""pjha"", ""timestamp"": ""1691669913.833899"", ""content"": ""<@U026PMDB1ND> I have launched RDS from the latest *collab* database snapshot in production. c.c <@U02BV2DGUKC>\nWriter instance *:* \nReader instance : ""}, {""user"": ""aganivada"", ""timestamp"": ""1691669990.540019"", ""content"": ""<@U04JT69T00K> does this instance use same password? Can we try accessing db with pgadmin or some client to make sure there is no issues connecting?""}, {""user"": ""pjha"", ""timestamp"": ""1691670251.070119"", ""content"": ""<@U02BV2DGUKC> yes, it uses same username and password, I tried accessing it from my Local terminal.""}, {""user"": ""aganivada"", ""timestamp"": ""1692161535.330679"", ""content"": ""cc: <@U026PMDB1ND>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692243683.101979"", ""content"": ""<@U02BV2DGUKC> - can you please remind me what we wanted to do here? I vaguely remember something about having a way in the app to update a timestamp in a record but don't remember much else""}, {""user"": ""aganivada"", ""timestamp"": ""1692244226.019509"", ""content"": ""sure <@U026PMDB1ND>, we were planning to have a button in retool to update current timestamp in any of the fields of collab db, we thought of using pulse submission time for testing. This change was planned directly from admin-gw to cloned collab db instance (details shared in thread).\n\nOnce we have this we could try executing db updates and see how applications are behaving (errors/timeout) during upgrade.\n\nretool integration is optional if we can make curl request it should suffice.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692253490.878289"", ""content"": ""<@U04JT69T00K> <@U02BV2DGUKC> - I'll do this tomorrow. Need to do a bit more than I originally thought it would take. Basically, we need to create read and write end points that talk to this DB and a few items on Retool that don't involve any of the existing screens.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692318211.301599"", ""content"": """"}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692318760.365289"", ""content"": ""Anil, if you leave the \""Pulse Name\"" empty and click \""Push and Pull\"" it would display the name currently in the pulse_survey table for that pulse_survey.\n\nIf you enter some text and push, it appends he current time, saves it in the db and updates the text field with the new value.\n\nIf you just keep clicking Push and Pull after that, it just keeps updating the timestamp portion of the message (so you can see that something is changing without having to type anything).\n\nThis is the current state of the record I have been playing with:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692318799.047449"", ""content"": ""It might be better if we can just play with this record and not touch much else.""}, {""user"": ""aganivada"", ""timestamp"": ""1692330253.113499"", ""content"": ""sure thank you <@U026PMDB1ND>""}, {""user"": ""aganivada"", ""timestamp"": ""1692330462.574529"", ""content"": ""<@U04JT69T00K> let me know when we plan to test upgrade I can verify operation from retool""}, {""user"": ""pjha"", ""timestamp"": ""1692334148.976059"", ""content"": ""sure <@U02BV2DGUKC>. We can plan it today sometime.""}, {""user"": ""pjha"", ""timestamp"": ""1692695343.431779"", ""content"": ""<@U026PMDB1ND> We have tested behaviour by upgrading engine version to 15.2.\n\u2022 There was Downtime, we could see 0 connection for around 20 minutes\n\u2022 Once the upgrade completed connection was established back without making any database url change in the application. \nc.c <@U02BV2DGUKC>""}, {""user"": ""pjha"", ""timestamp"": ""1692695473.835529"", ""content"": """"}, {""user"": ""pjha"", ""timestamp"": ""1692695618.762929"", ""content"": ""\nWe were getting 500 error during upgrade""}, {""user"": ""aganivada"", ""timestamp"": ""1692697820.413189"", ""content"": ""+ we didnt have to restart the services once database came up. Application was able to route connections automatically once the db was up""}, {""user"": ""aganivada"", ""timestamp"": ""1692698023.550799"", ""content"": ""<@U04JT69T00K> do you also want to cover the memory issue? is there any way we can check if the instances will not run into memory issue during upgrade?""}, {""user"": ""pjha"", ""timestamp"": ""1692699841.112699"", ""content"": ""<@U02BV2DGUKC> I have raised support ticket asking the memory configuration for smooth upgrade.""}]" "1692831799.469639 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1692831799.469639"", ""content"": ""<@U02BV2DGUKC> <@U03DHUAJVMK> <@U03NZ7Z52S2> - need your help....\n\nLooks like someone got the Stage Retro tenant into a weird state. For some unknown reason axm_retro_survey tenant (17203?) has become a tenant with no products. I need the data to prepare the Retro summary in the all hands meeting.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692831836.746189"", ""content"": ""cc <@U040RCBPBEC>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692832366.285879"", ""content"": ""The above is with .\n\nSatya tried to invite me () to this tenant, but I don't see the multiple tenant option after the login. It takes me to the platform retro directly.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692832399.868059"", ""content"": ""As if I cannot be invited to a new tenant.""}, {""user"": ""araman"", ""timestamp"": ""1692842231.207369"", ""content"": ""axm_retro_survey tenant (17203?) has become a tenant with no products.\n> I could see \""Retro Survey\"" engagement under 17203 <@U026PMDB1ND> ""}, {""user"": ""aganivada"", ""timestamp"": ""1692844725.975839"", ""content"": ""<@U026PMDB1ND> I invited as vendor to enable user actions. didnt know we were using this user, I closed the invite now operations should work.\n\n> but I don't see the multiple tenant option after the login. It takes me to the platform retro directly.\nthis is because () is associated with only one org so we dont show the option of selecting org to sign in to.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692846886.411239"", ""content"": ""~<@U02BV2DGUKC> - I still see the issue~""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692846948.050599"", ""content"": ""I am able to see the tenant details now""}, {""user"": ""aganivada"", ""timestamp"": ""1692846953.828109"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1692846955.580629"", ""content"": ""~can you try incognito?~""}]" "1690232889.947669 ","[{""user"": ""anair"", ""timestamp"": ""1690232889.947669"", ""content"": ""<@U02BV2DGUKC> <@U03KLHDKL1H> can we please add PacDen/17160 and Cisco/18096 to the snowflake dashboard.""}, {""user"": ""askumar"", ""timestamp"": ""1690260706.342979"", ""content"": ""Sure <@U033PPLNFRU>..will update it""}, {""user"": ""askumar"", ""timestamp"": ""1690265957.958989"", ""content"": ""Added the tenants.\nThanks""}]" "1687434888.524229 ","[{""user"": ""askumar"", ""timestamp"": ""1687434888.524229"", ""content"": "" <@U02BV2DGUKC>\nAdded AWS support ticket to help enable \""lo_compat_privileges\"" parameter for migration of blobs.\n""}, {""user"": ""askumar"", ""timestamp"": ""1687434918.324519"", ""content"": ""Request :\n\n\nWe are trying to migrate a schema from an RDS instance to Aurora RDS writer.\nHowever only the referenceId in the table is being migrated but the referred blob is not being migrated.\n\nWe check for specific flag and looks like there is paramater\nlo_compat_privileges\nthat is turned off and it control the access to the table pg_largeobject and\npg_largeobject_metadata.\n\nCan this be turned on, so that we can copy over the blobs.\n\nOr is there additional table mapping rules that we can update?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687446731.821239"", ""content"": ""<@U03KLHDKL1H> are there any known issues with turning it on?""}, {""user"": ""askumar"", ""timestamp"": ""1687447839.353019"", ""content"": ""<@U026PMDB1ND> This property used to be ON by default in previous versions\n\n\nThere can be compatibility issues with applications that rely on the default behavior/permissions, is what seems to be the implication (not 100% sure ). In this case since AWS RdsAdmin is managing access to the pg_catalog, so difficult to say what it could lead to.\n\nWhat I think it could be that applications reading the blobs indirectly using the Owner tables may face access issues.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687450651.444649"", ""content"": ""Do you need <@U02BV2DGUKC> or me to do something here or has this been taken care of?""}, {""user"": ""aganivada"", ""timestamp"": ""1687450723.432859"", ""content"": ""<@U026PMDB1ND> we had a discussion this afternoon, looks like aws is not allowing us to change the property so Ashwani raised the ticket. Once aws comes back we can check with them on alternative if they don't allow us to update the flag.""}, {""user"": ""ppant"", ""timestamp"": ""1687514845.582549"", ""content"": "" We tried to migrate the blobs using Python script (which you can find ) but this will not work as the HTML blobs for notification service does not cross the TOAST threshold. So if we do manual SQL insertion, then they are stored as is in the table.\n\nBut in the EmailTemplate JPA entity we have explicitly defined `htmlBody` to a . So any insertion of HTML templates via API forces Postgres to store it as a blob in `pg_catalog`. The only workaround is to call a brute force POST API to insert the templates""}, {""user"": ""aganivada"", ""timestamp"": ""1687516512.993869"", ""content"": ""<@U0431DZTPJM> if this is just for notification service can we not publish the templates again in int?""}, {""user"": ""aganivada"", ""timestamp"": ""1687516553.206539"", ""content"": ""Basically we can have empty templates table and run the publish job to upload all the templates""}, {""user"": ""ppant"", ""timestamp"": ""1687516620.127309"", ""content"": ""Publish job will also call the API of notification to insert the template, no?""}, {""user"": ""askumar"", ""timestamp"": ""1687516641.755909"", ""content"": ""It will be for token manager as well""}, {""user"": ""aganivada"", ""timestamp"": ""1687516698.026219"", ""content"": ""Ok but regarding the notification service how is this working in prod ?""}, {""user"": ""ppant"", ""timestamp"": ""1687516831.710589"", ""content"": ""In the table, in `htmlBody` column (which is text) IDs are stored which point to `pg_catalog.largeobject` . So when we call API to get the template, JPA automatically picks up the html body from pg_catalog (since we have already annotated that column with `@LoB` in entity definition)""}, {""user"": ""ppant"", ""timestamp"": ""1687516880.435809"", ""content"": ""Also the template insertion happens via API when we run the script to publish template. JPA in background breaks down the html string, stores it in pg_catalog and in `html` column stores the large object ID pointing to the table in pg_catalog""}, {""user"": ""aganivada"", ""timestamp"": ""1687517002.775649"", ""content"": ""hmmm still not clear why publishing templates via run-local.sh would fail""}, {""user"": ""aganivada"", ""timestamp"": ""1687517021.964539"", ""content"": ""can we get on a call? probably I am missing something""}, {""user"": ""askumar"", ""timestamp"": ""1687517040.527919"", ""content"": ""sure <@U02BV2DGUKC>""}, {""user"": ""ppant"", ""timestamp"": ""1687517047.421329"", ""content"": ""Oh I got confused. That should work""}, {""user"": ""askumar"", ""timestamp"": ""1687521092.242719"", ""content"": ""Migrated the notification templates with script.\nSo now notification service is also migrated to Aurora.""}, {""user"": ""askumar"", ""timestamp"": ""1687521205.167869"", ""content"": ""Thankyou <@U0431DZTPJM> <@U02BV2DGUKC>""}]" "1683673303.692929 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1683673303.692929"", ""content"": ""<@U02BV2DGUKC> <@U04JT69T00K> - did we change something with the Authorizers recently? With CDKv2 code, I was able to deploy services a couple of weeks back. Now, I get:\n\nUPDATE_ROLLBACK_COMPLETE: Resource handler returned message: \""Authorizer name must be unique. Authorizer lambda-authorizer-tagging-app already exists in this RestApi\n\nTried it in int and stage.""}, {""user"": ""aganivada"", ""timestamp"": ""1683689893.244859"", ""content"": ""I didn't make any changes <@U026PMDB1ND>, was holding my CDK changes for cdkv2. Ideally service should not be deploying authorizer again unless there are any changes""}, {""user"": ""aganivada"", ""timestamp"": ""1683690738.902949"", ""content"": ""<@U026PMDB1ND> can you share the synth output wanted to compare with cdkv1""}, {""user"": ""aganivada"", ""timestamp"": ""1683691021.790259"", ""content"": ""I see maximum authorizers reached error in logs. not sure why cdk was trying to create authorizer when there was already an existing authorizer for tagging-app""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683691704.320739"", ""content"": ""Will share""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683691726.839789"", ""content"": ""Interestingly I was able to deploy services with v2 a couple of weeks back""}, {""user"": ""aganivada"", ""timestamp"": ""1683691807.493309"", ""content"": ""hmm looking at tagging app's histroy looks like last deployment was a month back""}, {""user"": ""aganivada"", ""timestamp"": ""1683691813.785569"", ""content"": ""this is in int""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683691821.679419"", ""content"": ""Not tagging""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683691831.251809"", ""content"": ""Deployed a few other services""}, {""user"": ""aganivada"", ""timestamp"": ""1683691835.574719"", ""content"": ""ok""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683691878.114099"", ""content"": ""Did we update anything wrt authorizers for multi-instance work?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683691917.170839"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1683692003.514129"", ""content"": ""> Did we update anything wrt authorizers for multi-instance work?\nI dont think so <@U04JT69T00K> can you confirm? multi-intsance changes were not merged we decided to hold off merging to develop until we have cdkv2 changes done.""}, {""user"": ""pjha"", ""timestamp"": ""1683692164.238929"", ""content"": ""No, multi-instance changes are not merged. ""}, {""user"": ""aganivada"", ""timestamp"": ""1683692829.787699"", ""content"": ""<@U026PMDB1ND> didn't find any major change in synth output related to authorizer. Since there was an error on limit of authorizer in CF output I removed one of the authorizer mapping that we were not using in int. can we try deploy now?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683692856.425739"", ""content"": ""Where did you remove it, <@U02BV2DGUKC>?""}, {""user"": ""aganivada"", ""timestamp"": ""1683693081.177739"", ""content"": ""from authorizers <@U026PMDB1ND> removed a test sts-authorizer we added a while ago for testing it was not connected to any path""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683693811.470029"", ""content"": ""1308 tagging-app-stack-int | 9:39:31 PM | CREATE_FAILED | AWS::ApiGatewayV2::Authorizer | tagging-app-http-api/tagging-app-gateway-lambda-authorizer (taggingapphttpapitaggingappgatewaylambdaauthorizer1A53DD32) Resource handler returned message: \""Authorizer name must be unique. Authorizer lambda-authorizer-tagging-app already exists in this RestApi. (Service: AmazonApiGatewayV2; Status Code: 400; Error Code: BadRequestException; Request ID: b35953a0-8b55-428a-8a67-6c618dafdf5b; Proxy: null)\"" (RequestToken: a065e144-b680-f455-4193-d0938865d5a3, HandlerErrorCode: AlreadyExists )""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683693978.365309"", ""content"": ""How did we create these""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683693991.427689"", ""content"": ""specifically for the 4 apps""}, {""user"": ""aganivada"", ""timestamp"": ""1683694017.840849"", ""content"": ""They are created through cdk""}, {""user"": ""aganivada"", ""timestamp"": ""1683694037.997489"", ""content"": ""We add them manually only when we are testing something""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683694058.618169"", ""content"": ""Hmm.. basically cdkv2 is not seeing that an authorizer has already been added ?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683694099.715189"", ""content"": ""In my testing I hadn't deployed any apps services, I think. I might have just deployed core or vault""}, {""user"": ""aganivada"", ""timestamp"": ""1683694177.400499"", ""content"": ""ok that makes sense backend services dont use authorizers, there is one change we can do so CDK can re-use existing id but I am wondering why this is breaking in cdkv2 ideally it should have just updated the authorizer that is already deployed like in cdkv1""}, {""user"": ""aganivada"", ""timestamp"": ""1683694194.767069"", ""content"": ""let me take a look at the code changes for cdk v2""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683694794.453289"", ""content"": ""The metadata is different for the authorizer:\n\ncdk1: aws:cdk:path: tagging-app-stack-int/tagging-app-http-api/lambda-authorizer-tagging-app/Resource\n\ncdk2: aws:cdk:path: tagging-app-stack-int/tagging-app-http-api/tagging-app-gateway-lambda-authorizer/Resource""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683694815.473559"", ""content"": ""There seems to be a change on that line""}, {""user"": ""aganivada"", ""timestamp"": ""1683694849.071959"", ""content"": ""ohh ok may be that is why it was trying to add a new authorizer""}, {""user"": ""aganivada"", ""timestamp"": ""1683694914.462139"", ""content"": """"}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683694930.506599"", ""content"": ""Earlier, authorizers.HttpLambdaAuthorizer didn't need a name. Now they need a name.""}, {""user"": ""aganivada"", ""timestamp"": ""1683694985.911559"", ""content"": ""ok can we try updating name manually to see if that helps? worst case we might have to redeploy""}, {""user"": ""aganivada"", ""timestamp"": ""1683695067.380749"", ""content"": ""> didn't need a name\nwe were setting authorizer_name even before I think""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683695113.673599"", ""content"": ""Look at line 1311 in the previous screenshot""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683695120.796349"", ""content"": ""That is a v2 requirement""}, {""user"": ""aganivada"", ""timestamp"": ""1683695126.444949"", ""content"": ""ok got it""}, {""user"": ""aganivada"", ""timestamp"": ""1683695206.106899"", ""content"": ""did we change something? I see tagging-app update in progress""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683695261.675639"", ""content"": ""I basically adjusted the name in 1311 to match what was in cdkv1. But there's no guarantee that the update would be successful""}, {""user"": ""aganivada"", ""timestamp"": ""1683695271.200199"", ""content"": ""ok""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683695278.446819"", ""content"": ""It could go from update_progress -> update_rollback""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683695285.048939"", ""content"": ""most likely will""}, {""user"": ""aganivada"", ""timestamp"": ""1683695402.627889"", ""content"": ""lets see this time it went to ECS task""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683695428.432729"", ""content"": ""It completed :man-facepalming:""}, {""user"": ""aganivada"", ""timestamp"": ""1683695437.285939"", ""content"": "":grinning:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683695449.584409"", ""content"": ""Maybe reverse psychology worked :stuck_out_tongue:""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683695463.661199"", ""content"": ""Or I am talking to machines like I talk to my kids""}, {""user"": ""aganivada"", ""timestamp"": ""1683695572.086419"", ""content"": "":grinning: that change actually makes sense because earlier we had metadata name and authorizer_name different. May be they use metadata to figure out if they need to add new resource""}, {""user"": ""aganivada"", ""timestamp"": ""1683695602.925249"", ""content"": ""we can update both to new naming convention but we have a limit of 10 authorizers to a api-gw and we might run out of that limit""}, {""user"": ""aganivada"", ""timestamp"": ""1683695637.277699"", ""content"": ""we'd be ok if they automatically delete the old one""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683696102.012109"", ""content"": ""> that change actually makes sense because earlier we had metadata name and authorizer_name different. May be they use metadata to figure out if they need to add new resource\nYeah, that is how they seem to do it.\n\nWas not documented anywhere""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683696143.216969"", ""content"": ""Maybe we need to deploy everything once (say over the weekend) before merging to develop""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683696156.703409"", ""content"": ""This kind of scares me a bit""}, {""user"": ""aganivada"", ""timestamp"": ""1683696288.563139"", ""content"": ""sure, we can divide between ourselves and try deploying all apps services and test. We can also deploy some basic core stacks but some core stacks like api-gateway/rds might be a bit risky.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683696328.559299"", ""content"": ""I think that we can deploy core stacks as needed""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683696483.895219"", ""content"": ""If we can deploy the service stacks (apps, backend, lambdas, state-machine etc) we know that we can move to v2. If we need to deploy a core stack, we'll probably be able to do it at that time.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683696539.950999"", ""content"": """"}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683696558.738259"", ""content"": ""You can use this to try v2.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683696612.537929"", ""content"": ""Will require a bit of tinkering to get both v1 and v2 running on the same box, but it is doable. I did it just now""}, {""user"": ""aganivada"", ""timestamp"": ""1683696637.122199"", ""content"": ""cool, will try to set it up and during weekend we can deploy all services""}]" "1677555250.894129 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1677555250.894129"", ""content"": ""<@U040RCBPBEC> <@U026PMDB1ND> <@U02BV2DGUKC> - i got access to clarity - internal monitoring tool for imply - got access for stage and int clusters- i have asked for production cluster access - The interesting thing is they have concept called alert - like in lightstep- which supports webhook - we can send to our slack channels - will create a jira ticket to create alerts for all environments""}, {""user"": ""aganivada"", ""timestamp"": ""1677556109.538189"", ""content"": ""cool thank you <@U0336QZAF98>, do they support multiple destinations so that we can also forward alert to opsgenie? not sure if opsgenie supports webhook though""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1677560472.515759"", ""content"": ""<@U02BV2DGUKC> - they support Custom - where you can populate payload - Do you know if opsgenie supports this format ?""}, {""user"": ""aganivada"", ""timestamp"": ""1677563795.811579"", ""content"": ""not sure <@U0336QZAF98>, we can verify""}, {""user"": ""aganivada"", ""timestamp"": ""1677564251.662459"", ""content"": ""<@U026PMDB1ND> can you add me or <@U0336QZAF98> as admin temporarily to verify if we can hookup imply to opsgenie""}]" "1684944658.529099 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1684944658.529099"", ""content"": ""Respond/Answer etc make more sense to me. Complete is an action that can show up after a \""Start\"", IMO.\n\n<@U04EV0M2VA6> <@U033PPLNFRU> does \""Complete Pulse\"" make sense to you as a starting action?""}, {""user"": ""araman"", ""timestamp"": ""1684945089.339839"", ""content"": ""as far i remember, we did not previous or back button for slack""}, {""user"": ""anair"", ""timestamp"": ""1684952078.397389"", ""content"": ""<@U02TVMF3CR4> can you please update the copy to \""Start Pulse\"". cc <@U026PMDB1ND>""}, {""user"": ""mli"", ""timestamp"": ""1684962204.795069"", ""content"": ""Agreed <@U026PMDB1ND> we were discussing this when working on new Gamification designs. We want to change the CTA from \u201cComplete Pulse\u201d to \u201cStart Pulse\u201d, especially because we will introduce the ability to complete a half-finished Pulse where the CTA will be \u201cResume Pulse\u201d.\n\n<@U02TVMF3CR4> can we please also change the CTA on the FINAL Pulse page from \u201cNext\u201d to \u201cSubmit\u201d? Next implies that there are more questions to come. cc <@U03ML44ADTQ>""}, {""user"": ""mli"", ""timestamp"": ""1684962209.973959"", ""content"": """"}, {""user"": ""mnirmal"", ""timestamp"": ""1684994877.242689"", ""content"": ""Sure <@U04EV0M2VA6> <@U033PPLNFRU>, I'll update the slack CTA. JFYI, this was specifically called out when I posted a demo during the development phase - .""}, {""user"": ""mli"", ""timestamp"": ""1685082633.235899"", ""content"": ""Noted, sorry for missing this earlier""}]" "1685104264.759009 ","[{""user"": ""nsrivastava"", ""timestamp"": ""1685104264.759009"", ""content"": "" one of the Jobs in SC got abruptly terminated with log `ERROR: Job failed (system failure): aborted: terminated`\n\n""}, {""user"": ""nsrivastava"", ""timestamp"": ""1685104365.549969"", ""content"": ""Also the re-run the in queue, as of now, not sure if this is manually killed""}, {""user"": ""aganivada"", ""timestamp"": ""1685105205.175929"", ""content"": ""hmmm <@U04JT69T00K> saw a similar behaviour once with SC job is this some config issue?""}, {""user"": ""aganivada"", ""timestamp"": ""1685105217.676689"", ""content"": ""<@U03RQDE3QUS> looking into it""}, {""user"": ""aganivada"", ""timestamp"": ""1685105782.548729"", ""content"": ""<@U03RQDE3QUS> pipeline started now\n\nPrashant I think we need to replicate the scenario there was a sudden spike in CPU""}, {""user"": ""aganivada"", ""timestamp"": ""1685106116.532969"", ""content"": ""<@U03RQDE3QUS> jfyi one test failed I triggered verify again just to check the CPU spike please ignore""}, {""user"": ""pjha"", ""timestamp"": ""1685120061.437369"", ""content"": ""<@U02BV2DGUKC> Instance did't stop seems not to be responding. I will enhance monitoring for the Instance.""}, {""user"": ""aganivada"", ""timestamp"": ""1685120131.720289"", ""content"": ""we need to set alerts on this instance when we run short of resources.""}, {""user"": ""pjha"", ""timestamp"": ""1685120321.249139"", ""content"": ""sure I will configure it""}, {""user"": ""pjha"", ""timestamp"": ""1685120391.082659"", ""content"": ""<@U02BV2DGUKC> Just wanted to check did you change the webhook configuration for sc ?""}, {""user"": ""aganivada"", ""timestamp"": ""1685120726.596599"", ""content"": ""Yes <@U04JT69T00K>""}, {""user"": ""aganivada"", ""timestamp"": ""1685120756.606319"", ""content"": ""I mean I did configure webhook for sc""}]" "1692263935.187779 ","[{""user"": ""aganivada"", ""timestamp"": ""1692263935.187779"", ""content"": ""<@U02GC8SE18V> <@U03NZ7Z52S2> backend changes for updated invitation flow is deployed in int. FF for this feature is turned on for all tenants by default in int""}, {""user"": ""bganganna"", ""timestamp"": ""1692264500.168809"", ""content"": ""<@U02BV2DGUKC> is this related to open invitation flow changes ?""}, {""user"": ""aganivada"", ""timestamp"": ""1692264607.401339"", ""content"": ""Yes <@U03NZ7Z52S2> the in-flight invitation flow. Now we should get only one invitation instead of multiple invitations everytime user is added to a engagement and hasn't onboarded yet. Also removing a user from engagement should not result in deleting invitation.""}, {""user"": ""aganivada"", ""timestamp"": ""1692264694.012589"", ""content"": ""After onboarding we need to invoke a api call for now manually (I shared a demo loom link in slack 1-1). Once we have apps an UI changes this flow should be automated""}, {""user"": ""aganivada"", ""timestamp"": ""1692274977.781549"", ""content"": ""<@U02GC8SE18V> jfyi 404 is an expected response for getStatus, if the user does not have any open invitations or has an invitation that was added before we deployed this feature we might just return 404 in which case no need to call update status. We can call update status only when getStatus returns 200 with response of onboarded false.""}, {""user"": ""rvaidya"", ""timestamp"": ""1692679756.401809"", ""content"": ""<@U02BV2DGUKC> the app change for this is deployed in INT, incase you would like to test it. cc <@U03NZ7Z52S2>""}, {""user"": ""aganivada"", ""timestamp"": ""1692679798.685559"", ""content"": ""thank you <@U02GC8SE18V>, will take a look""}, {""user"": ""aganivada"", ""timestamp"": ""1692692297.113399"", ""content"": ""<@U02GC8SE18V> once the user logs in we are having to refresh the page for all the permissions to show up can we automate page reload after closing membership? cc: <@U02HCMTQU3W>\n\n""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1692693351.401219"", ""content"": ""<@U02BV2DGUKC> Can you elaborate what you mean? Are we missing something from UI flow? How did you add permissions to this user?""}, {""user"": ""rvaidya"", ""timestamp"": ""1692693522.621899"", ""content"": ""Sure <@U02BV2DGUKC>.\n<@U02HCMTQU3W> As part of login, we are triggering a flow in tenancy to complete permission on apps. Post login, looks like UI need to refresh the page. (refresh token and call /user/info after /login). Should be doable ?""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1692694429.646869"", ""content"": ""Its doable <@U02GC8SE18V>, yes we need a flag for this in info call to let ui know that we need to reload the tokens cc <@U04ARFN3WQP>""}, {""user"": ""aganivada"", ""timestamp"": ""1692694700.190439"", ""content"": ""thank you <@U02GC8SE18V>,\n\n<@U02HCMTQU3W> I will add a UI ticket with the details Ritu mentioned above""}, {""user"": ""rvaidya"", ""timestamp"": ""1692694736.974449"", ""content"": ""<@U02HCMTQU3W> <@U04ARFN3WQP> Currently this is flow of API calls at login time :\n\nI think what i am asking is : org > token > login > token > info instead of org > token > info > login""}, {""user"": ""bot_message"", ""timestamp"": ""1692699841.257789"", ""content"": ""@Pradeep created a Task UI-2605 Refresh Token based on flag from info call""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1692699861.473869"", ""content"": ""<@U02GC8SE18V> created tracker we can do this once we have a flag sent from info call""}, {""user"": ""aganivada"", ""timestamp"": ""1693830302.708269"", ""content"": ""<@U03NZ7Z52S2> jfyi is qa-ready\n\n<@U02HCMTQU3W> /<@U04ARFN3WQP> is this change deployed to int?""}, {""user"": ""bganganna"", ""timestamp"": ""1693894996.895169"", ""content"": ""<@U02HCMTQU3W> <@U04ARFN3WQP> looks like build is not deployed ,still seeing org > token > info > login order""}, {""user"": ""amishra"", ""timestamp"": ""1693895282.895979"", ""content"": ""<@U03NZ7Z52S2> We did not change the API order instead, we re-fetched the token and info call if inviteMerged is false.\ncc: <@U02GC8SE18V>""}, {""user"": ""bganganna"", ""timestamp"": ""1693895579.933249"", ""content"": ""<@U04ARFN3WQP> inviteMerged is ff or api response ?""}, {""user"": ""amishra"", ""timestamp"": ""1693895599.343119"", ""content"": ""api response in info call""}]" "1683147383.857809 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1683147383.857809"", ""content"": ""<@U02BV2DGUKC> <@U04JT69T00K> - now that 0.9.10 is deployed, have we finished the release? We have to bring the CDKv2 code into develop, but to be safe, we can start a HF branch (after release is finished) and then merge the cdkv2 to develop.\n\nThat way we can continue to deploy off of that HF branch while we try things out with CDKv2.\n\nAny thoughts?""}, {""user"": ""aganivada"", ""timestamp"": ""1683172347.784939"", ""content"": ""<@U026PMDB1ND> we finished the and a also has been added to cdk-artifacts for next deployment. But it looks like Mohith committed from local to release/0.9.10 I will update him to move to HF branch instead and delete the release/0.9.10. We should be good to push cdkv2 changes to develop.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683174109.598329"", ""content"": ""Let me know after Mohith moves his code and the release branch is finished ""}, {""user"": ""aganivada"", ""timestamp"": ""1683174461.441699"", ""content"": ""sure Rama""}, {""user"": ""aganivada"", ""timestamp"": ""1683215963.516589"", ""content"": ""<@U02SF36PVKL> please post here once the changes are moved from release/0.9.10 to HF branch we will delete release/0.9.10 branch""}, {""user"": ""mnirmal"", ""timestamp"": ""1683222860.372719"", ""content"": ""<@U02BV2DGUKC> updated the new SC version in hotfix/0.9.10-rel1 - you can delete this branch""}, {""user"": ""aganivada"", ""timestamp"": ""1683259478.452769"", ""content"": ""cool thank you <@U02SF36PVKL>""}, {""user"": ""aganivada"", ""timestamp"": ""1683259908.010529"", ""content"": ""<@U026PMDB1ND>\n1. Release/0.9.10 is deleted now\n2. HF branch created and informed teams to use this for all HF deployments . \n3. Tags published for 0.9.10 to cdk-artifacts project - this should help us in case both main and develop run into issues after cdkv2 changes""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683259953.222189"", ""content"": ""cool. thanks ""}]" "1677130766.792109 ","[{""user"": ""rvaidya"", ""timestamp"": ""1677130766.792109"", ""content"": ""<@U0336QZAF98> just wanted to understand if the notification service supports attachments (like PDF) to be sent along with emails ?\ncc <@U028EDANJM9> <@U02SCRTM2M7> <@U02HCMTQU3W>""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1677135079.408599"", ""content"": ""<@U02GC8SE18V> yes it is supported - both inline attachments (base64 encoded data) as well as s3 path - For s3 - notification service reads from a predetermined bucket - only ask is - you have to write the file in that bucket and pass the path - Below are system test cases for the same\n""}]" "1688449571.430299 ","[{""user"": ""pjha"", ""timestamp"": ""1688449571.430299"", ""content"": "" Please review RDS Proxy Document ""}, {""user"": ""aganivada"", ""timestamp"": ""1688452435.903649"", ""content"": ""thank you <@U04JT69T00K> did we enable this in int or some env? cc: <@U026PMDB1ND>""}, {""user"": ""pjha"", ""timestamp"": ""1688452516.192229"", ""content"": ""I have configured it for test Database, not enabled to the actual one.""}, {""user"": ""aganivada"", ""timestamp"": ""1688452566.998879"", ""content"": ""can we configure for int DB? may be we can check if we are getting the stats we are looking for""}, {""user"": ""askumar"", ""timestamp"": ""1688452908.791749"", ""content"": ""cool <@U04JT69T00K>\nalso, do we need to recreate the RDS secrets for this, by specifying the DB ?""}, {""user"": ""pjha"", ""timestamp"": ""1688453110.812509"", ""content"": ""<@U03KLHDKL1H> I will try adding the DB information to the SecretValue(*/int/rds-secret-common*) if that works then we don't need to create, else we might have to create with the same name.""}, {""user"": ""pjha"", ""timestamp"": ""1688465455.626459"", ""content"": ""<@U02BV2DGUKC> <@U03KLHDKL1H> Proxy for the \t'aslv2-common-int'\nEndpoint : /aslv2-common-int-proxy-read-only.endpoint.proxy-cm7fc5ggz2fk.us-west-2.rds.amazonaws.com\n\nFor now I have created new secret for proxy to communicate with the database.""}, {""user"": ""aganivada"", ""timestamp"": ""1688465696.594799"", ""content"": ""<@U04JT69T00K> would new secret create an issue while configuring services to use proxy?""}, {""user"": ""pjha"", ""timestamp"": ""1688466877.710679"", ""content"": ""<@U02BV2DGUKC> no it won't create any issue but we will have to manage two secret, better if we have single secret.\n\nHere as we can see in '/int/rds-secret-common' we only have 'db_user' and 'db_password' whereas in '/int/rds-secret-common-proxy' we have additional database information, we can anyway add additional information to the '/int/rds-secret-common' but here we have 'db_user'/'db_password' instead of 'username'/'password' which we use in code to connect to the database, therefore either we have to modify the code or have one more secret.""}]" "1686264405.265519 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1686264405.265519"", ""content"": "" Had I not taken the day off yesterday and closed my previous sprint tasks before the retro that I missed, we'd have hit the highest per capita in the last 15 weeks (I was sitting on 30 points) :disappointed:""}, {""user"": ""aganivada"", ""timestamp"": ""1686284087.771399"", ""content"": ""174 points Nice!! :star-struck: I think prev. highest was 160+ less than a month ago\n\nwe have a new goal to beat :slightly_smiling_face:""}]" "1681908421.968619 ","[{""user"": ""askumar"", ""timestamp"": ""1681908421.968619"", ""content"": ""\n\nInitial schema design for storing events info in snowflake, that has both the basic info fields and a JSON fields that can store all the event info.\n\n<@U02BV2DGUKC> <@U02SF36PVKL>\nCurrently I have populated events table with around 3k records for the pulse events such as :\nPulse Sent, Button clicked, Submitted\nThis is based on the the class POJOs that we have for pulse data.\n\nPlease have a look.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1681909971.493189"", ""content"": ""<@U03KLHDKL1H> - Main table - i can see that it is tightly coupled with pulse related event - snowflake has very good support for json - root columns can be one on one with EventFramework schema - like event_id (guid) , event_type_ event_created_at , event_received_at , tenant_id - rest of other columns you can try to put it inside data part of it""}, {""user"": ""askumar"", ""timestamp"": ""1681910154.217309"", ""content"": ""sure <@U0336QZAF98>.. we will modify accordingly""}, {""user"": ""askumar"", ""timestamp"": ""1681911779.350359"", ""content"": ""Added the alternate base table as well to snowflake and doc.""}]" "1672907416.886919 ","[{""user"": ""akasim"", ""timestamp"": ""1672907416.886919"", ""content"": ""\nToday when I did hotfix-finish, the merge happened from hf branch to develop and main branches successfully. But the pipeline build didn't kicked-in automatically for both develop and main. Is this a known issue or did I do something wrong?\nProject : \nhf: hotfix/0.0.13.4\n_Note: Surprisingly the hf branch was not deleted from git._\n<@U02BV2DGUKC> <@U026PMDB1ND> <@U02D4DUKDQC>\nScreenshot from IDE:""}, {""user"": ""sranjan"", ""timestamp"": ""1672907492.757329"", ""content"": ""Same happened to me as well but on feature branch.""}, {""user"": ""akasim"", ""timestamp"": ""1672907496.649229"", ""content"": ""There is a warning in screenshot w.r.t remote branch deletion. Is that the reason for not triggering the pipeline?""}, {""user"": ""pkarthikeyan"", ""timestamp"": ""1672907600.131599"", ""content"": ""I am also facing the same issue.""}, {""user"": ""sfarooqh"", ""timestamp"": ""1672908248.772089"", ""content"": ""same thing i observed in feature branch""}, {""user"": ""aganivada"", ""timestamp"": ""1672908341.546529"", ""content"": ""checking <@U02HQ78V9A5>""}, {""user"": ""aganivada"", ""timestamp"": ""1672908377.906069"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1672908406.196479"", ""content"": ""gitlab is having outage""}, {""user"": ""aganivada"", ""timestamp"": ""1672908501.559769"", ""content"": ""<@U02HQ78V9A5> / <@U03BPNY5AGM> / <@U02SCRTM2M7> / <@U034RLJA97X> apart from HF branch not getting deleted and builds not being triggered (hopefully should trigger once gitlab recovers) is there any irrecoverable issue ?""}, {""user"": ""sfarooqh"", ""timestamp"": ""1672908553.537639"", ""content"": ""<@U02BV2DGUKC> i did not observe anything else apart from that""}, {""user"": ""akasim"", ""timestamp"": ""1672908608.049689"", ""content"": ""nothing <@U02BV2DGUKC> thats it.""}, {""user"": ""aganivada"", ""timestamp"": ""1672908659.565709"", ""content"": ""ok thank you, can we trigger pipeline manually? I can help if there is any permissions related issue""}, {""user"": ""akasim"", ""timestamp"": ""1672908839.345729"", ""content"": ""I am able to trigger pipeline manually. Thanks <@U02BV2DGUKC>""}]" "1687246118.740619 ","[{""user"": ""sranjan"", ""timestamp"": ""1687246118.740619"", ""content"": "" <@U02BV2DGUKC> Plz bounce MB in stage . cc <@U03NZ7Z52S2>""}, {""user"": ""aganivada"", ""timestamp"": ""1687249153.171209"", ""content"": ""<@U034RLJA97X> should be done in another 5-10 mins trigerred bounce of MB in stage""}]" "1676565328.500489 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1676565328.500489"", ""content"": ""Not the exact name, but a part of the name, for instance. Do we have an API?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1676571039.557389"", ""content"": ""visibility v2 search on tenancy - ""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1676571060.516499"", ""content"": ""name=pin=anu%""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1676571171.585889"", ""content"": ""<@U026PMDB1ND>\n```\nIf you want to search on multiple regex\nname=pin=('%test%','%anu%') ```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1676571744.408589"", ""content"": ""For now, I just implemented a script which does =ilike=<given string> and serves the purpose.\n\nWhat does \""pin\"" stand for btw?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1676571778.110799"", ""content"": ""Partial in operator - which takes array of pattern""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1676571805.649949"", ""content"": ""Cool.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1676571808.482779"", ""content"": ""Thanks""}]" "1688566935.678079 ","[{""user"": ""aganivada"", ""timestamp"": ""1688566935.678079"", ""content"": ""<@U04JT69T00K> can we try pulling latest develon of cdk to multi-instance branch and resolve conflicts?""}, {""user"": ""aganivada"", ""timestamp"": ""1688566955.416399"", ""content"": ""Please let me know once it is done I have a few changes to be tested""}, {""user"": ""pjha"", ""timestamp"": ""1688567056.005019"", ""content"": ""Yes we can pull latest develop branch changes ""}, {""user"": ""aganivada"", ""timestamp"": ""1688567120.294339"", ""content"": ""<@U04JT69T00K> I tried to pull and got some conflicts, dont want to break the multi-instance changes so can you check when you get a chance?""}, {""user"": ""pjha"", ""timestamp"": ""1688567200.754119"", ""content"": ""Sure, I will check. Currently I am away is it fine if I resolve it in sometime?""}, {""user"": ""aganivada"", ""timestamp"": ""1688567233.278849"", ""content"": ""sure no hurry, let me know once you get a chance to look into it""}, {""user"": ""pjha"", ""timestamp"": ""1688567240.284079"", ""content"": ""Sure""}]" "1694015610.021479 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1694015610.021479"", ""content"": "" - I am getting three notifucations -Thanks for providing feedback for completing survey -for submitting one pulse - Is it intended? ""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1694017999.508839"", ""content"": ""Can you post a screen grab <@U0336QZAF98> as the link is just showing only one message. ~I believe it may be intended~""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1694021149.294559"", ""content"": ""Here is screenshot <@U02D4DUKDQC>""}, {""user"": ""mnirmal"", ""timestamp"": ""1694021313.968979"", ""content"": ""<@U0336QZAF98> did you use remind me later functionality or can you confirm if you were nudged from the UI? We generally edit all the nudge messages we send to this message once the pulse has been completed.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1694021476.294469"", ""content"": ""No\u2026i filled it the very first time i got notification- new pulse has been created - not sure if i got any nudege <@U02BV2DGUKC> - did you nudge me from UI""}, {""user"": ""aganivada"", ""timestamp"": ""1694021527.618329"", ""content"": ""I did nudge users a few times from dashboard <@U02SF36PVKL>""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1694021540.278919"", ""content"": ""I did not use Remind me <@U02SF36PVKL>""}, {""user"": ""aganivada"", ""timestamp"": ""1694021542.970429"", ""content"": ""<@U0336QZAF98> I dont recollect if your name was in list""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1694021560.406619"", ""content"": ""yeah i fiilled it immediately""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1694022288.185639"", ""content"": ""Thanks for looking into it <@U02SF36PVKL>""}, {""user"": ""mnirmal"", ""timestamp"": ""1694023098.704179"", ""content"": ""<@U02BV2DGUKC> would be very helpful if we can just confirm if the instance id is 134994""}, {""user"": ""mnirmal"", ""timestamp"": ""1694024182.082329"", ""content"": ""<@U02BV2DGUKC> <@U02D4DUKDQC> can you please share the result for\n\n```select * from svc_sentiment_collector_default_axm.scheduled_user_nudge where LOWER(user_id) = '' and pulse_survey_instance_id=134994;```\n""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1694025345.055499"", ""content"": ""<@U02SF36PVKL>\n\n```id,tenant_id,pulse_survey_instance_id,user_id,service,nudge_time_in_utc,is_nudged,message_id,nudge_type\n139547,15328,134994,rsrinivasan@axiamatic.com,slack,2023-09-06 14:04:22.900270 +00:00,true,1694009063.048579,PO_INITIATED\n139799,15328,134994,rsrinivasan@axiamatic.com,slack,2023-09-06 15:45:05.692563 +00:00,true,1694015105.839309,PO_INITIATED\n139807,15328,134994,rsrinivasan@axiamatic.com,slack,2023-09-06 15:47:01.601944 +00:00,true,1694015221.710629,PO_INITIATED\n139815,15328,134994,rsrinivasan@axiamatic.com,slack,2023-09-06 15:48:56.341867 +00:00,true,1694015336.432589,PO_INITIATED```""}, {""user"": ""mnirmal"", ""timestamp"": ""1694052946.887479"", ""content"": ""Thanks <@U02D4DUKDQC>. <@U0336QZAF98> your response time acc to logs is \""Sep 6, 2023 @ 21:20:46.646\"" and <@U02BV2DGUKC> nudged you at around Sept 6 19:30 and 21:15.\n\nThe count is also in-line with the records we have - I can see 4 records and 4 messages in your screenshot.""}, {""user"": ""mnirmal"", ""timestamp"": ""1694053025.963949"", ""content"": ""So basically we're not sending \""Thanks for providing feedback for completing survey\"", we are just editing the already sent nudge messages.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1694054491.215619"", ""content"": ""<@U02SF36PVKL> - If there are multiple nudges , should n\u2019t we update only the last/latest one or send a completely new message that its been completed - instead of updating all four ?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1694054552.158239"", ""content"": ""i got four notifications in a very short span of time""}, {""user"": ""mnirmal"", ""timestamp"": ""1694070014.919269"", ""content"": ""<@U0336QZAF98> I was trying to search the thread were we decided to update all the messages of the nudge, but couldn't find it and lost track of it.\n\n<@U0336QZAF98> also, wrt your on getting them in short span of time - I think thats how anil triggered it. You can see the time at which <@U02BV2DGUKC> triggered the nudges -\n1. 14:04:22.900270 UTC\n2. 15:45:05.692563 UTC\n3. 15:47:01.601944 UTC\n4. 15:48:56.341867 UTC\nAs you can see the last 3 were triggered in a span of few mins (this you can see in your screenshot as 3 hours ago) and first was sent an hour before those 3 (seen in screenshot as 2 hours ago)""}, {""user"": ""aganivada"", ""timestamp"": ""1694071323.048839"", ""content"": ""Agreed, we were trying a few flows so have triggered nudge a few times to check. I can confirm these are manually triggered nudges.""}, {""user"": ""aganivada"", ""timestamp"": ""1694071419.146849"", ""content"": ""> I was trying to search the thread were we decided to update all the messages of the nudge\n<@U02SF36PVKL> if we are going with this approach can we update in knowledge base ""}]" "1681964605.549509 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1681964605.549509"", ""content"": ""My son is not well. So i have to take him to hospital. Will connect in the evening""}, {""user"": ""aganivada"", ""timestamp"": ""1681964631.163559"", ""content"": ""take care <@U0336QZAF98>""}]" "1680755519.784839 ","[{""user"": ""aganivada"", ""timestamp"": ""1680755519.784839"", ""content"": ""<@U0431DZTPJM> /<@U0336QZAF98> setup 3-4pm IST today to syncup on running SSO migration script in prod. If we cant cover setup and migration today we can split between today and monday. Goal is to replicate tenant similar to lvsands (roles+permissions) and run migration script instructions.""}, {""user"": ""aganivada"", ""timestamp"": ""1680783045.560209"", ""content"": ""<@U0336QZAF98>, <@U0431DZTPJM> updated the code to skip non-existing user let me know when you are free we can continue""}]" "1685429383.901229 ","[{""user"": ""nsrivastava"", ""timestamp"": ""1685429383.901229"", ""content"": "" facing issue while uploading consuming Slack-Integrator snapshot dependency\n\n\nthe job mentions building the artefact `slack-integrator-0.0.5-SNAPSHOT: digest: sha256:7c4ad984fa9e828f9c33e92f9d86ea275585efcb0b90719eeb57c25870893160 size: 6375` but it is not present in package registry \nor I am looking at wrong place\nthe SC MR is failing due to dep not found.\n.\n\nCan someone please look into this""}, {""user"": ""aganivada"", ""timestamp"": ""1685430193.632999"", ""content"": ""<@U03RQDE3QUS> quick call?""}, {""user"": ""nsrivastava"", ""timestamp"": ""1685434441.935759"", ""content"": ""<@U02BV2DGUKC> the snapshots seem to get published fine in core \n\nthe registry has version `0.0.17-COLL-2498-SNAPSHOT` that I just cut from develop. but same was not the case with slack-integrator for this MR version `0.0.5-COLL-2498-1-SNAPSHOT`""}, {""user"": ""aganivada"", ""timestamp"": ""1685434566.771249"", ""content"": ""That's weird, <@U03RQDE3QUS> can we trigger slack-integrator pipeline again? May be it was a temporary issue in gitlab""}, {""user"": ""nsrivastava"", ""timestamp"": ""1685434622.918749"", ""content"": ""sure <@U02BV2DGUKC> triggered ""}, {""user"": ""nsrivastava"", ""timestamp"": ""1685437250.815289"", ""content"": ""the issue is fixed. Main had few commits that got added as build fix during release and were missing from dev, syncing dev with main helped resolving.""}]" "1674771653.249369 ","[{""user"": ""anair"", ""timestamp"": ""1674771653.249369"", ""content"": ""<@U02BV2DGUKC> <@U02D4DUKDQC> for the monitoring piece instead of days, weeks, months could we actually measure in from start date to end date""}, {""user"": ""aganivada"", ""timestamp"": ""1674786255.676049"", ""content"": ""<@U033PPLNFRU> metrics systems are typically counters so I am not sure if they fit in start and end date. let me review and get back. Can you share some sample queries you might be interested in ? ""}, {""user"": ""aganivada"", ""timestamp"": ""1674786268.954789"", ""content"": ""cc: <@U02SF36PVKL> <@U03KLHDKL1H>""}, {""user"": ""anair"", ""timestamp"": ""1674806684.141149"", ""content"": ""<@U02D4DUKDQC> <@U02BV2DGUKC>\n\n*I think what we have with pulses rn is sufficient, could we surface pulse instance ID so can take an easy look up using swagger.* \n\nHere are some statements I had re: actions + messaging. Please let me know which are feasible + if you think we should some. In terms of priority if we can deliver on either \""between\"" or \""in the last\"" to begin with that would be great.\n\nActions\n1. {customer name} created action {action name} of type {action type} for {vendor name} on {date}\n2. {vendor name} created action {action name} of type {action type} for {customer name} on {date}\n3. Between {start date} and {end date}, {customer name} created {n} actions for {vendor name}\n4. Between {start date} and {end date}, {vendor name} created {n} actions for {customer name}\n5. In the last {week/2weeks/month}, {vendor name} created {n} actions for {customer name}\n6. In the last {week/2weeks/month}, {customer name} created {n} actions for {vendor name}\nMessaging\n1. Between {start date} and {end date} there were {n} messages sent by {customer name} to {vendor name} \n2. Between {start date} and {end date} there were {n} messages sent by {vendor name} to {customer name} \n3. Replace between ..... --> in the last {week/2weeks/month}\nMessaging with user details\n1. Between {start date} and {end date} there were {n} messages sent by {customer 1...customer n} from {customer name} to {vendor name} \n2. Between {start date} and {end date} there were {n} messages sent by {vendor user 1...vendor user n} from {vendor name} to {customer name} \nEverything else (reports, stakeholders, collaborators) is lower priority and can be pushed back""}, {""user"": ""aganivada"", ""timestamp"": ""1674815472.541689"", ""content"": ""got it thank you Aryan, I think most of the cases where we have customer and vendor dimensions we dont have the data in grafana will check with <@U02SF36PVKL>if we can add these dimensions. We might be able to get some of this info from logz however the limit in logz is currently 14 days whereas in grafana we have retention upto 13 months""}, {""user"": ""anair"", ""timestamp"": ""1674854379.947189"", ""content"": ""<@U02SF36PVKL> <@U02D4DUKDQC> <@U02BV2DGUKC> we really need to pick this up with the highest priority""}, {""user"": ""aganivada"", ""timestamp"": ""1675049605.337929"", ""content"": ""<@U02SF36PVKL> scheduling a call today to discuss on metrics, I guess we need to add a few more attributes to support the metrics""}, {""user"": ""aganivada"", ""timestamp"": ""1675068678.775899"", ""content"": ""cc: <@U02D4DUKDQC>""}]" "1681242136.026399 ","[{""user"": ""aganivada"", ""timestamp"": ""1681242136.026399"", ""content"": ""<@U033PPLNFRU> <@U0336QZAF98> one option in SSO we could explore for automatic membership if customers add to a group is auth0's auto membership config. Basically users can be provided access to app and when they login they will automatically be added to the organization. However when they login they may see an error page since they are not assigned a role. once orgadmin assigns a role they will be able to access the app. The advantage here is admins won't have to add user and send invitation via Axiamatic.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1681243516.493119"", ""content"": ""<@U02BV2DGUKC> Makes sense . If we combine with this - then we can create rule which assign users automatically to any role based on configuration\n""}, {""user"": ""aganivada"", ""timestamp"": ""1681243624.864419"", ""content"": ""yup""}, {""user"": ""aganivada"", ""timestamp"": ""1681243654.197389"", ""content"": ""assigning default role + this setting should address the ask.""}, {""user"": ""anair"", ""timestamp"": ""1681275296.998279"", ""content"": ""can we just assign them org admin by default for now? Is this a config change that can be easily done? I think for now just assigning everyone as org admin could be easier for now <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1681277602.935459"", ""content"": ""sure <@U033PPLNFRU> we can do that from auth0""}, {""user"": ""aganivada"", ""timestamp"": ""1681277740.927689"", ""content"": ""<@U033PPLNFRU> assigned orgadmin role to (SSO user) other users are already orgadmins""}, {""user"": ""aganivada"", ""timestamp"": ""1681277780.008569"", ""content"": ""can we delete the invitations, even if Mathew clicks on open invitation it should not cause any issues since his role is already orgadmin""}]" "1685947152.321359 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1685947152.321359"", ""content"": ""<@U02BV2DGUKC> <@U04JT69T00K> - i am getting not enough space error on axiamtic gitlab runner - - could you help me out here ?\n```Caused by: org.testcontainers.containers.ContainerFetchException: Failed to get Docker client for \nCaused by: com.github.dockerjava.api.exception.DockerClientException: Could not pull image: write /var/lib/docker/tmp/GetImageBlob1469629156: no space left on device```""}, {""user"": ""pjha"", ""timestamp"": ""1685947839.474249"", ""content"": ""<@U0336QZAF98> please check now""}, {""user"": ""aganivada"", ""timestamp"": ""1685948251.316029"", ""content"": ""<@U04JT69T00K> how much space do we have now?""}, {""user"": ""pjha"", ""timestamp"": ""1685948294.379089"", ""content"": ""21 % is used, Avail is 93G""}, {""user"": ""aganivada"", ""timestamp"": ""1685948352.257409"", ""content"": ""Ok seems decent what was the size earlier? May be we should have an alert if capacity meets say 60%""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1685948424.057649"", ""content"": ""<@U04JT69T00K> - does it keep increasing over for every build - if possible we can see what is adding up to space after every build - most probably it should be docker - we need to see how to we can delete docker images locally after every build""}, {""user"": ""pjha"", ""timestamp"": ""1685948439.555409"", ""content"": ""earlier used was 114G out of 117G.""}, {""user"": ""pjha"", ""timestamp"": ""1685948440.821079"", ""content"": ""<@U02BV2DGUKC> yes, I have assigned a task to myself to enhance monitoring for gitlab""}, {""user"": ""pjha"", ""timestamp"": ""1685948463.371849"", ""content"": ""<@U0336QZAF98> dangling images was there and also unused container volumes.""}, {""user"": ""aganivada"", ""timestamp"": ""1685962087.348889"", ""content"": ""<@U04JT69T00K> are we also planning to cleanup unused containers as part of the task?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1685963171.336199"", ""content"": ""<@U04JT69T00K> - thank you - i am able to generate build and deploy""}, {""user"": ""pjha"", ""timestamp"": ""1685985669.497289"", ""content"": ""<@U02BV2DGUKC> yes will schedule cron job to clean unused images and volumes""}]" "1687334230.646369 ","[{""user"": ""pjha"", ""timestamp"": ""1687334230.646369"", ""content"": "" please review Deployment for lambda function using tag ""}, {""user"": ""aganivada"", ""timestamp"": ""1687348671.427539"", ""content"": ""thank you <@U04JT69T00K> added some comments please review. Can we demo this during SOS?""}, {""user"": ""pjha"", ""timestamp"": ""1687348761.462719"", ""content"": ""Sure <@U02BV2DGUKC> , I will go through the comments. Yes we can demo it . ""}, {""user"": ""pjha"", ""timestamp"": ""1687411186.148249"", ""content"": ""<@U02BV2DGUKC> I addressed your comments.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687478316.003579"", ""content"": ""<@U04JT69T00K> - Can we tag the feature branch versions also with 'SNAPSHOT'?\n\nAlso, is there any pre-commit hook script that we can add which would ensure that versionflow is in place?\n\nAre you planning to update .gitlab-ci.yml in all the lambda repos in one shot?\n\nBasically, it would be nice to have a clear idea of how this idea is going to be implemented across the board.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687478433.815039"", ""content"": ""Can we avoid apt-get installs during build and move them into the base image?""}]" "1684318623.949609 ","[{""user"": ""askumar"", ""timestamp"": ""1684318623.949609"", ""content"": ""\nPlease have a look at the general migration plan document.\nSome of the links/images may get updated as we move ahead with deployments.\n\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684425681.316289"", ""content"": ""I'll review this today""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684538049.055849"", ""content"": ""Looks good, <@U03KLHDKL1H>.\n\n<#C0453KXLJRZ|> please take a look as this is going to impact all the services. Thanks!""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684538097.509859"", ""content"": ""<@U03KLHDKL1H> - how exactly would the liquibase tables be merged?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684538182.262539"", ""content"": ""How long do we estimate the whole of INT migration to complete?""}, {""user"": ""askumar"", ""timestamp"": ""1684573201.633969"", ""content"": ""<@U026PMDB1ND>\nFor Liquibase tables, since the structure is similar , one table will be created.\nEach service can copy entries of their schema into that table.\nSince liquibase is checking against the checksum, if we keep same checksum , then entries can coexist.\nI have not yet verified this liquibase this merge for multiple services, will do this.\n\n\nFor time span, 2 releases I am expecting at least.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684729308.069559"", ""content"": ""<@U03KLHDKL1H> - each service copying its own data is not going to work. We need to figure out a way to do this in one shot.""}, {""user"": ""aganivada"", ""timestamp"": ""1684729701.941069"", ""content"": ""<@U03KLHDKL1H> based on our prev discussions I thought with pgdump we can update/insert only the records that are missing in db. so when we move a new service cant we take a dump of exiting integration db and apply on aurora cluster?""}, {""user"": ""askumar"", ""timestamp"": ""1684733085.943089"", ""content"": ""<@U026PMDB1ND> To avoid asking each service to copy liquibase table data, we can prepopulate the liquibase table and when they are migrating we can just copy the delta. Will it help?\n\nFor schema/table data they will need to take a back up and do a restore ,if we need to keep data in sync.""}, {""user"": ""askumar"", ""timestamp"": ""1684733282.605269"", ""content"": ""<@U02BV2DGUKC> The pgdump works differently, since it has to copy the entire database, it works by creating tables , populating entries of data and then adding constraints.\n\nI am not sure if it can just copy the delta given the steps it follows.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684857013.401889"", ""content"": ""<@U03KLHDKL1H> - it makes sense in general. I don't understand \""copying the delta\"" part""}, {""user"": ""askumar"", ""timestamp"": ""1685339110.684489"", ""content"": ""<@U026PMDB1ND> sorry for replying so late, I was caught in a personal exigency while on vacation.\n\nFor liquibase table :\nIf we copy the entries from the current RDS to Aurora entries right now, then each service won't have to do copy the liquibase table data.\n\nBy delta, I meant that if there is any additional liquibase entry that might be there in RDS when service is migrated from RDS to Aurora, we will copy that additional entry, before migration is started.\n\nPlease let me know your ideas on this.""}]" "1680581305.085279 ","[{""user"": ""aganivada"", ""timestamp"": ""1680581305.085279"", ""content"": "" planning to spend 2-3 days next sprint only on refactoring tenancy, overtime this has become unmanageable (code + formatting)""}, {""user"": ""ppant"", ""timestamp"": ""1680581346.101839"", ""content"": ""Let me know if you need help. I am done with the lambda coding (development going on), happy to pick this up""}, {""user"": ""aganivada"", ""timestamp"": ""1680618568.482589"", ""content"": ""Sure thank you <@U0431DZTPJM>""}]" "1683614386.558849 ","[{""user"": ""ppant"", ""timestamp"": ""1683614386.558849"", ""content"": "" Please use the following dependencies in Poetry. On 3rd May, Python\u2019s requests rolled out a new version 2.30.0 which made `boto3=1.26.93` that we were using, incompatible with `urllib3 < 1.26.5` (we use 1.26.15). Earlier we had added `requests = \""^2.27.1\""` in Poetry which got auto updated in new pipelines causing runtime errors. The version of requests that works with all our dependencies is `2.28.2`\n\n```[tool.poetry.dependencies]\npython = \""^3.8\""\nrequests = \""2.28.2\""\nplatform-libs-epoch = {version = \""0.0.8\"", source = \""axmcommon\""}\n\n[tool.poetry.dev-dependencies]\nbandit = \""1.7.5\""\nboto3 = \""1.26.93\""\nbotocore = \""1.29.115\""\nblack = { version = \""23.1.0\"", allow-prereleases = true }\ncoverage = \""7.2.2\""\nflake8 = \""5.0.4\""\nipython = \""8.11.0\""\nisort = { version = \""5.12.0\"", extras = [\""pyproject\""] }\njinja2-cli = \""0.8.2\""\nmkdocs = \""1.3.0\""\nmkdocstrings = \""0.20.0\""\nmkdocs-material = \""8.5.4\""\nsafety = \""2.3.4\""\npylint = \""2.17.0\""\npytest = \""7.2.2\""\npytest-cov = \""4.0.0\""\npytest-sugar = \""0.9.6\""\npytest-xdist = \""3.2.1\""\nurllib3 = \""1.26.15\""```""}, {""user"": ""aganivada"", ""timestamp"": ""1683644834.386089"", ""content"": ""<@U0431DZTPJM> please post this in <#C028U328HMG|engg> channel""}]" "1685681026.535339 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1685681026.535339"", ""content"": ""We actually have checked this code in because we had to modify something a tiny bit. We can see if there's a way to allow SELECT to work""}, {""user"": ""aganivada"", ""timestamp"": ""1685682328.275509"", ""content"": ""got it, will take a look at ""}]" "1684944832.532149 ","[{""user"": ""bganganna"", ""timestamp"": ""1684944832.532149"", ""content"": ""Looks like this was missed as part of pulse card improvements <@U026PMDB1ND>. cc <@U02SF36PVKL>""}, {""user"": ""mnirmal"", ""timestamp"": ""1684945276.464299"", ""content"": ""what was missed <@U03NZ7Z52S2>? \""Complete Pulse\"", is part of the spec. Earlier it was \""Start Pulse\""""}, {""user"": ""bganganna"", ""timestamp"": ""1684945758.811709"", ""content"": ""back button for slack survey <@U02SF36PVKL>""}, {""user"": ""bganganna"", ""timestamp"": ""1684945909.257739"", ""content"": ""Sorry my bad, i think in teams we have previous button may be not in slack <@U02SF36PVKL> <@U026PMDB1ND>""}]" "1678246833.928209 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1678246833.928209"", ""content"": "" - something funky \""seems\"" to be going on with GitLab maven today. Added a new library and referenced it from core-data. The builds pass fine locally. The lib passes on GitLab. When I clean up the lib's artifacts on .m2/repository and run the core-data build again, I see that they get downloaded from GitLab again and the build pass just fine.\n\nBut when I run the core-data build on GitLab, it says there's an access forbidden error in fetching the lib. If I remove the reference to the lib the build goes through fine.\n\nNot sure if I am made a mistake in the setting up the lib, but if I did, I don't see why I can pull the lib locally. Strange stuff ate up almost 4 hours of my time today :(""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678246978.962599"", ""content"": ""Did anyone run into an issue like this earlier?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678247039.527679"", ""content"": ""401 unauthorised i got couple of times this week - I go to .M2/settings.xml - i update to new token""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247061.905969"", ""content"": ""On your machine?""}, {""user"": ""aganivada"", ""timestamp"": ""1678247062.249999"", ""content"": ""this issue seems to be on gitlab""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678247069.692059"", ""content"": ""Yes on my machine""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247073.805289"", ""content"": ""Yeah.. it is the reverse issue.""}, {""user"": ""aganivada"", ""timestamp"": ""1678247079.787879"", ""content"": ""<@U026PMDB1ND> did you try clearing runner cache?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247086.610129"", ""content"": ""Works fine locally. Breaks on GitLab.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247093.894819"", ""content"": ""Yeah.. cleaned the cache""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247121.084069"", ""content"": ""I even created a new project because this was driving me crazy""}, {""user"": ""aganivada"", ""timestamp"": ""1678247153.225769"", ""content"": ""hmm this is weird what is the library? I can try hooking up to someother project""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247153.604019"", ""content"": ""Did mvn -U as well""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247166.784459"", ""content"": ""The issue is with authorization. Not with just the cache""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247224.972519"", ""content"": ""> I can try hooking up to someother project\n\nI did that too. Added to a service that has no real dependence on the lib. Same issue.""}, {""user"": ""aganivada"", ""timestamp"": ""1678247251.456289"", ""content"": ""ok""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678247257.288199"", ""content"": ""In gitlab - what is authorization model to pull libs - is it role based or dynamic token ?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247278.391759"", ""content"": ""Opened a case with them.. but I am not sure if there's something I messed up""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247294.226039"", ""content"": ""dynamic token""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247312.662469"", ""content"": ""The job is able to pull every other lib. Only this new lib is the problem""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247348.265599"", ""content"": ""So, the token is valid""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247359.764769"", ""content"": ""Or at least valid for everything else""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678247361.483989"", ""content"": ""Got it...which project this is ?""}, {""user"": ""aganivada"", ""timestamp"": ""1678247374.021709"", ""content"": ""> This failure was cached in the local repository and resolution is not reattempted until the update interval of gitlab-maven has elapsed or updates are forced. Original error: Could not transfer artifact com.axm.platform:axm-auditlog-name-resolution:pom:0.0.1-SNAPSHOT from/to gitlab-maven""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247388.987499"", ""content"": ""platform-libs-name-resolution""}, {""user"": ""aganivada"", ""timestamp"": ""1678247391.933129"", ""content"": ""this seems to be hapening even after -U""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247464.019439"", ""content"": ""The referring service is platform-services-auditlog (which doesn't really need it) or core-services-data that needs it.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247548.315999"", ""content"": ""That cached error was happening when the name of the lib was \""platform-libs-auditlog-name-resolution\"". Created a new one and the first pull fails with \""authorization error\"" which will be saved in the cache.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247604.145469"", ""content"": ""What a horrible waste of time.""}, {""user"": ""aganivada"", ""timestamp"": ""1678247882.301609"", ""content"": ""added -e command for detailed logs and now we are getting this exception \n\n> Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: The following artifacts could not be resolved: com.axm.platform:axm-auditlog-name-resolution:jar:0.0.1-SNAPSHOT, com.axm.platform.services:core-data-specification:jar:0.0.14-PLAT-1620-SNAPSHOT: Could not find artifact com.axm.platform:axm-auditlog-name-resolution:jar:0.0.1-SNAPSHOT in gitlab-maven ()""}, {""user"": ""aganivada"", ""timestamp"": ""1678247952.907859"", ""content"": ""this is weird may be it is a runner issue, we can try on the ec2 runner""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247970.576219"", ""content"": ""Anil, that one is still referring to the old lib""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678247985.173099"", ""content"": ""will change it now""}, {""user"": ""aganivada"", ""timestamp"": ""1678247989.576009"", ""content"": ""ohh ok""}, {""user"": ""aganivada"", ""timestamp"": ""1678248001.905669"", ""content"": ""this is the branch Rama ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678248148.340469"", ""content"": ""right..""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678248235.282829"", ""content"": ""A few more days like this and I am going to slink off into a premature retirement and spend my remaining days in poverty :slightly_smiling_face:""}, {""user"": ""aganivada"", ""timestamp"": ""1678248722.698869"", ""content"": ""same error not so helpful message :disappointed:""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678248774.733579"", ""content"": ""me 2""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678248775.066809"", ""content"": ""Caused by: org.apache.maven.wagon.authorization.AuthorizationException: authorization failed for , status: 403 Forbidden\n\n""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678248828.569869"", ""content"": ""it says 403 ..mine was 401 - either the token could not see the new project for some reason - <@U026PMDB1ND> did u move project from private to axiamatic - something like that""}, {""user"": ""aganivada"", ""timestamp"": ""1678248846.357569"", ""content"": """"}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678248990.483709"", ""content"": ""> did u move project from private to axiamatic\nNo.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678249024.168639"", ""content"": ""> \nI guess it makes sense if the problem is with the local build, right?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678249132.336109"", ""content"": ""Completed 5.5 hours on this crap. I am going to wait for a response from GitLab support. They hopefully know WTF is going on.""}, {""user"": ""aganivada"", ""timestamp"": ""1678249150.361679"", ""content"": ""Yeah thats is correct ideally they should generate a token which has sufficient permissions in runner""}, {""user"": ""aganivada"", ""timestamp"": ""1678249220.879509"", ""content"": ""~may be we can raise a bug/ticket in gitlab~""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678305469.718479"", ""content"": ""Can't believe this, Apparently GitLab released a new \""feature\"" which forces the owner of a project to list out all the projects that are allowed to access the artifacts of the current project. There are no wild-cards either.\n\nI also saw the attached screenshot along the way. Not sure if they considered how people release libraries in a Java shop. Fricking scary.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678305890.018139"", ""content"": ""Tagging <@U040RCBPBEC>""}, {""user"": ""aganivada"", ""timestamp"": ""1678334932.897289"", ""content"": ""<@U026PMDB1ND> is this not impacting other existing libraries or is runner getting libraries from cache?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678343491.066579"", ""content"": ""Only the new libraries get created with this flag turned on, <@U02BV2DGUKC>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678343537.626919"", ""content"": ""I ended up being the Axm guinea pig for their roll-out :slightly_smiling_face:""}, {""user"": ""aganivada"", ""timestamp"": ""1678343815.375639"", ""content"": ""> Only the new libraries get created with this flag turned on\nthis is weird, they don't have same policy applied across libraries? Not sure the reasoning behind marking it as default for new libraries when existing libraries are not being updated.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678344016.113619"", ""content"": ""There's a note that appears on the screen that by May 2024 this needs to be done for all the projects.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678344061.804559"", ""content"": ""So, maybe they'll turn this on for all the projects at the time and cause a bunch of builds to fail.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678344069.461889"", ""content"": ""Weid fellas""}, {""user"": ""svummidi"", ""timestamp"": ""1678377828.566059"", ""content"": ""I think it is important to understand the security implications, as per their documentation, they are claiming it as a serious security risk to disable this feature. I don\u2019t understand the security concern described in the below sentence.\n> It is a security risk to disable the allowlist. A malicious user could try to compromise a pipeline created in an unauthorized project. If the pipeline was created by one of your maintainers, the job token could be used in an attempt to access your project.\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678377945.644329"", ""content"": ""Will forward the thread""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678394019.200929"", ""content"": ""The support guy dropped the question I asked about this, so asked him again.\n\nEarlier he said that it is to restrict people within Axiamatic Main because outsiders wouldn't even be able to access. If that's correct and we'd saving ourselves from a malicious user in Axiamatic Main who can do a lot more damage than transitively downloading a maven dependency :slightly_smiling_face: OTOH, if that blurb is completely true, then everything is broken today not just newly created projects.\n\nLet's see what they come back with.""}, {""user"": ""svummidi"", ""timestamp"": ""1678394966.085139"", ""content"": ""Yes, in their statement, if both `malicious user` and `maintainer` coming from our organization then it is not a real issue.""}]" "1693477195.981009 ","[{""user"": ""aganivada"", ""timestamp"": ""1693477195.981009"", ""content"": ""<@U02HCMTQU3W> <@U04ARFN3WQP> <@U04JT69T00K> following are the points I noted from platform perspective for UI stack deployment changes:\n\n1. Add a new s3 bucket and link to cloud front stack - What is the name we should use for this domain? (we are currently using app-ui) once the domain is available we might have to cname the domain to \n2. Make changes to CDK cloud front to add some additional options - <@U02HCMTQU3W> to share details\n3. Finalize on CICD model - We will be generating tar files instead of images. Is it ok to continue with existing model of Gitlab -> ECR account during publish & ECR account -> AWS (int/stage/prod) during deployment? <@U04JT69T00K> to evaluate (research spike)\n4. Hydrate env.js during deployment before uploading to s3 - As of today we are doing this when container starts\n5. Update UI CDK deployment code to support both models and gradually stop using container model \n6. Add command to invalidate cloudfront cache after publishing content to s3 bucket - <@U04ARFN3WQP> to share the command\n7. Add cloud watch alarms for error responses from cloudfront logs\n8. (If possible) Add checksum validation at the end of deployment - <@U04ARFN3WQP> /<@U02HCMTQU3W> to share details on this\n9. Evaluate process to cleanup stale files in s3\ncc: <@U026PMDB1ND>""}, {""user"": ""aganivada"", ""timestamp"": ""1693477734.958379"", ""content"": ""Added for this""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693494981.834999"", ""content"": ""What is the main driver to go away from the container model?""}, {""user"": ""aganivada"", ""timestamp"": ""1693495126.974349"", ""content"": ""<@U026PMDB1ND> this change is for speeding up our UI and serving directly from cloudfront so we don't need any ecs instance""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1693495584.166429"", ""content"": ""<@U026PMDB1ND> here is the demo from <@U04ARFN3WQP> on advantages hosting using cloudfront over hosting in ECS.\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693496212.912849"", ""content"": ""Thanks <@U02HCMTQU3W>""}, {""user"": ""amishra"", ""timestamp"": ""1693496234.135299"", ""content"": ""Stats link - \n<@U026PMDB1ND>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693496345.983899"", ""content"": ""Nice. Thanks <@U04ARFN3WQP>""}, {""user"": ""aganivada"", ""timestamp"": ""1693537791.308969"", ""content"": ""<@U04JT69T00K> please review tasks under and add sprint \""PLAT Sprint 110 - 0.9.15 #4\"" to tasks we can target this sprint cc: <@U02HCMTQU3W>""}, {""user"": ""aganivada"", ""timestamp"": ""1693537826.169109"", ""content"": ""<@U02HCMTQU3W> just for my confirmation is this change targeted to 0.9.15 or 0.9.16?""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1693540927.781989"", ""content"": ""<@U02BV2DGUKC> we can target 0.9.16. This is driven more from tech debt and benefits in user experience that we see from this. we did not get any prioritization on this from product.\n\ncc <@U033PPLNFRU> <@U04NPHNJMCP>""}]" "1680041665.863659 ","[{""user"": ""svummidi"", ""timestamp"": ""1680041665.863659"", ""content"": "" <@U033PPLNFRU> <@U03ML44ADTQ> <@U0336QZAF98> - Some of the topics generated in our Retro tenant emerged from the question. Do you feel it is a concern? Do we need to exclude the Topics from the question to pick the popular Topics?""}, {""user"": ""anair"", ""timestamp"": ""1680041993.989699"", ""content"": ""<@U040RCBPBEC> should we cap the number of topics to the number of responses? Does each topic have a relevance score or something like that?\n\nAlso how did the word \""drop\"" get in there? Was it from the question text?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680042036.578279"", ""content"": ""Can't see when would it make sense to add words from the question to the topic list.""}, {""user"": ""svummidi"", ""timestamp"": ""1680042377.660509"", ""content"": ""<@U033PPLNFRU> - We are showing popular topics - If a Topic is referred from multiple users, that is considered as more popular.\nIf we include Topics from the question, they always take the first row. It looks like it is a problem.\nI created a ticket to track this ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680043590.265589"", ""content"": ""What happens if we only use the answer and not include the question text for topic analysis, <@U040RCBPBEC>?""}, {""user"": ""svummidi"", ""timestamp"": ""1680046411.922249"", ""content"": ""<@U026PMDB1ND> We are making one query to OpenAI to get both sentiment and topics. If we make separate calls it will double our cost. For sentiment detection adding question context is helping but it is impacting the topics. Probably we need to do some GPT prompt engineering so we can establish the context about Question and Answer and ask model to exclude topics from the question.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680046515.604189"", ""content"": ""<@U040RCBPBEC> what is the cost in $ terms for each call?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680046780.849249"", ""content"": ""Maybe we can do this for tenants who we white-list so that we don't waste money on everything, while also not sacrificing efficacy.\n\nIf we drop all the terms from the question from the final list, we may also drop some key terms from the answers, if they exist in both.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680046909.659489"", ""content"": ""In this case, if we had a major issue with message 'drop' in MB and we remove 'drop' because it is in the question, it could distort the topic list.""}, {""user"": ""svummidi"", ""timestamp"": ""1680051098.370489"", ""content"": ""<@U026PMDB1ND> - Rough estimate for current cost @1.50 for 1K comments.\nAs mentioned in my last comment, I think we can solve this problem by asking the model to exclude questions for topic generation instead of excluding topics after they generated.\nIf nothing works we can consider two pass approach.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1680075348.271129"", ""content"": ""<@U040RCBPBEC> - Typically topics are picked from questions when the answer is empty or N/A or irrelevant. Will work on prompt so that topics are picked only from answer""}, {""user"": ""avenkataraman"", ""timestamp"": ""1680081449.761979"", ""content"": ""Yes <@U040RCBPBEC>, the primary objective of introducing the topics was for PO's to view and filter the top issues. Having the questions words/Text will not help the PO, we need the topics generated only from the answers. This is seen as a method for summarisation.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1680108242.724119"", ""content"": ""<@U040RCBPBEC> - i regenerated topics for 17203 - With modified prompt - for the same tenant i dont see any topic from question""}, {""user"": ""svummidi"", ""timestamp"": ""1680109394.068909"", ""content"": ""<@U0336QZAF98> - Great! Thank you so much for quickly fixing this. Please update for Prod and default in code also.""}, {""user"": ""svummidi"", ""timestamp"": ""1680110016.464419"", ""content"": ""Below is the screenshot after fixing the prompt - Now the topics looking good. Even though there is one topic with \u201cRequirement Clarity\u201d actually it came up from the actual answer.\n<@U03ML44ADTQ> and <@U033PPLNFRU> Please check this Dogfood tenant and let us know if there is anything odd pops up for sentiment detection or topics.\n<@U03ML44ADTQ> - Not sure about your comment on summarization. Few possible ways to summarize the retro are listed in the below post. Let us know if you have suggestions on the approach for summarization.\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680110110.482469"", ""content"": ""The topics look good now :clap:""}, {""user"": ""anair"", ""timestamp"": ""1680111181.464839"", ""content"": ""Thanks <@U040RCBPBEC> looks great, <@U0336QZAF98> what new prompt did you run? Is this ?""}, {""user"": ""avenkataraman"", ""timestamp"": ""1680143405.319969"", ""content"": ""Thanks <@U040RCBPBEC>: Looks very insightful""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1680149297.152839"", ""content"": ""<@U033PPLNFRU> - For gpt3.5 - below link points to the new prompt - ""}]" "1683888514.387719 ","[{""user"": ""mnirmal"", ""timestamp"": ""1683888514.387719"", ""content"": "" I can't find the deployment dependencies page for 0.9.11. Can someone please share? Or please let me know if I should be creating one.""}, {""user"": ""aganivada"", ""timestamp"": ""1683899227.913609"", ""content"": ""<@U02SF36PVKL> will verify\n\n<@U04JT69T00K> can we check if there is a deployment dependency page added for 0.9.11 and add one if it is added yet. Please use latest deployment dependency template""}, {""user"": ""pjha"", ""timestamp"": ""1683899251.417469"", ""content"": ""sure, I will check""}, {""user"": ""pjha"", ""timestamp"": ""1683899729.402669"", ""content"": ""<@U02SF36PVKL> Deployment dependencies page c.c <@U02BV2DGUKC>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683921501.101659"", ""content"": ""Thanks <@U04JT69T00K>""}]" "1689259888.626099 ","[{""user"": ""aganivada"", ""timestamp"": ""1689259888.626099"", ""content"": ""<@U03KLHDKL1H> can we shutdown commons rds in stage?""}, {""user"": ""askumar"", ""timestamp"": ""1689261376.854049"", ""content"": ""Sure <@U02BV2DGUKC> I will remove it""}, {""user"": ""askumar"", ""timestamp"": ""1689261603.521789"", ""content"": ""<@U02BV2DGUKC> deletion of instance should be fine now, we have backed up the data and there are no more connections to it ?""}, {""user"": ""aganivada"", ""timestamp"": ""1689261631.533249"", ""content"": ""yup""}, {""user"": ""askumar"", ""timestamp"": ""1689261647.243019"", ""content"": ""cool...Thankyou""}, {""user"": ""askumar"", ""timestamp"": ""1689262244.749569"", ""content"": ""Removed the commons RDS instances.""}]" "1693381316.844789 ","[{""user"": ""psomasamudram563"", ""timestamp"": ""1693381316.844789"", ""content"": ""<@U02BV2DGUKC> we are running $300 billing in the AWS Core Account, can we shut down a few things if we can?""}, {""user"": ""aganivada"", ""timestamp"": ""1693392084.228949"", ""content"": ""Sure <@U0281D3GWHL>""}, {""user"": ""aganivada"", ""timestamp"": ""1693400925.931589"", ""content"": "" jfyi deleting all the zones added to devx account ""}, {""user"": ""aganivada"", ""timestamp"": ""1693402094.024859"", ""content"": ""<@U0281D3GWHL> all the hosted zones have been deleted, and bastion host shutdown. Sorry about the delay.""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1693409130.596299"", ""content"": ""Thanks <@U02BV2DGUKC>""}]" "1690564803.511329 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1690564803.511329"", ""content"": ""<@U02BV2DGUKC> - do we know if anyone makes calls to the epoch service?\n<@U02GC8SE18V> - should we take down tagging?""}, {""user"": ""aganivada"", ""timestamp"": ""1690564856.261909"", ""content"": ""Not sure <@U026PMDB1ND> will take a look and update by tmrw""}, {""user"": ""askumar"", ""timestamp"": ""1690565776.007219"", ""content"": ""<@U026PMDB1ND> the only invocation of epoch is from tenant deletion workflow in logs.\nThere are no new epoch workflows instances being created/used.""}, {""user"": ""rvaidya"", ""timestamp"": ""1690773725.363959"", ""content"": ""Yes <@U026PMDB1ND>, thats the plan with 0.9.14 deployment.\nSince we didnot had a formal 0.9.13 to prod, we will be retiring tagging app in 0.9.14.\ncc <@U03ML44G5RC> <@U028EDANJM9>""}]" "1646873474.414469 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1646873474.414469"", ""content"": ""<@U02BV2DGUKC>, in the authorizer attached to the API GW do we look for the presence of a S2S token before deciding to admit the traffic? Does it have something configured to let the Swagger and health URLs pass through without a token check?\n\nWe'd have to update some of this to make sure the calls are coming in from the associated VPN for Swagger and health end points. The health end point change would require us to remove the health check end points from system tests run from GitLab.""}, {""user"": ""aganivada"", ""timestamp"": ""1646887956.177159"", ""content"": ""<@U026PMDB1ND> Authorizer is enabled only on app services and currently authorizer does not understand s2s token. for the services where custom authorizer is enabled we have whitelisted swagger-urls this is being done today by adding a separate route (non proxy) and removing authorizer \n\n\n> def whitelist_authorizer_calls(self, http_api: IHttpApi):\n> \n> for whitelist in onboarding_app_constants.Params.WHITELIST_ENDPOINTS.resolve(self.envw):\n> (whitelist)\n> (whitelist['endpoint'])\n> self.svc_api_gateway_route = api_gw.HttpRoute(\n> self,\n> \""{}-api-gateway-route-for-{}\"".format(self.params.service_name, whitelist['endpoint']),\n> http_api=http_api,\n> route_key=api_gw.HttpRouteKey.with_(path=whitelist['endpoint'], method=HttpMethod[whitelist['method']]),\n> integration=self.svc_gateway_integration,\n> ) \nI think we can update custom authorizer to allow calls from an associated VPN.\n\nFor system tests made from gitlab can we add a ip range that gitlab will be using while making calls?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1646894491.708289"", ""content"": ""For the calls from GitLab, we could just tag the health calls and exclude them. So, we'd basically not need to check health from GitLab at all. All other API calls with use S2S.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1646894644.766369"", ""content"": ""1. For apps services, we should update the authorizer to to allow swagger and health for connections from a VPN (btw, I have not tested any of this).""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1646894708.036029"", ""content"": ""2. For non-apps, I think we should have a pass through authorizer (if possible) that would allow the calls only from the VPN.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1646894721.903409"", ""content"": ""Can we do something like this?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1646894807.464359"", ""content"": ""I guess we don't need an authorizer.. ""}, {""user"": ""aganivada"", ""timestamp"": ""1646896823.726349"", ""content"": ""> Can we do something like this?\nshould be possible, custom authorizer does get source ip address, but api-gw route seems a lot better""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1646933243.202909"", ""content"": ""yup""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1672893177.978089"", ""content"": ""<@U02BV2DGUKC> - blast from the past :slightly_smiling_face:""}, {""user"": ""aganivada"", ""timestamp"": ""1672894574.180639"", ""content"": ""<@U026PMDB1ND> we did evaluate , this suggestgion was for api-gw rest-api while we use http-api based on this suggestion we built ip-authorizer lambda which would basically allow or deny based on ip range that comes from vpn. sorry for not updating this thread.""}]" "1685590689.881599 ","[{""user"": ""aganivada"", ""timestamp"": ""1685590689.881599"", ""content"": ""Congratulations <@U0431DZTPJM> & <@U04JT69T00K>!! :clap::tada:""}, {""user"": ""ppant"", ""timestamp"": ""1685599611.621379"", ""content"": ""<@U02BV2DGUKC> So when are you coming to Bangalore to give us a party for this? :smile:""}, {""user"": ""aganivada"", ""timestamp"": ""1685600003.682969"", ""content"": "":grinning: as soon as I get a chance :slightly_smiling_face:""}]" "1692891034.145749 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1692891034.145749"", ""content"": ""<@U02BV2DGUKC> - did you connect with <@U033PPLNFRU> re: 'Unbound' users Auth0?""}, {""user"": ""aganivada"", ""timestamp"": ""1692891372.531509"", ""content"": ""yes <@U026PMDB1ND> , we enabled the ff now and fixed one of the users who had both username-pwd and saml enabled for the other users with username pwd, I will share report with Aryan he said he will take a look and decide if we need to re-invite the users""}, {""user"": ""aganivada"", ""timestamp"": ""1692891919.413369"", ""content"": ""<@U033PPLNFRU> <@U026PMDB1ND> here is the list\n\nEA ->\n\nUsername-password : to be reinvited so they can join via SAML\n1. - product admin\n2. - orgadmin\n3. - orgadmin\n4. - orgadmin\n\n\nSAML users:\n1. \n2. \n\n\nPacden -> all customer users joined via SAML so no re-invites required, issue we ran into with was probably because we had both username password and SAML enabled""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692895122.656889"", ""content"": ""<@U02BV2DGUKC> - the issue was primarily not turning on the flag before initiating SSO?""}, {""user"": ""aganivada"", ""timestamp"": ""1692895367.137509"", ""content"": ""no <@U026PMDB1ND> issue happened because we missed turning on the flag which forces all new invitations to go via SAML once we were done with SSO. we did tenant level config to route invitations via SAML but there was another FF that was controlling the flow. We were doing this initially when we were configuring SSO but somewhere we missed this step for EA and pacden. Now we enabled this FF to true in prod after testing in stage ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692895768.586289"", ""content"": ""<@U02BV2DGUKC> - do we need to update the page which has instructions for enabling SSO to talk about this flag as well?""}, {""user"": ""aganivada"", ""timestamp"": ""1692895908.716079"", ""content"": ""sure <@U026PMDB1ND> will do, we are anyways planning to remove this FF. I had a question to you on this do you recollect why we marked as not required?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692896044.047629"", ""content"": ""If a flag is marked as permanent after we create the ticket. At least that's the intent :slightly_smiling_face:""}, {""user"": ""aganivada"", ""timestamp"": ""1692896089.949139"", ""content"": ""ahh I see thank you <@U026PMDB1ND>. not sure why we marked this as permanent. I think we can remove it will check with Seshan""}, {""user"": ""aganivada"", ""timestamp"": ""1692896180.211789"", ""content"": ""<@U0336QZAF98> <@U026PMDB1ND> <@U0431DZTPJM> please review updated instructions to check FF status. once we remove the FF we can delete this check""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692896477.920369"", ""content"": ""<@U02BV2DGUKC> - now it is no longer marked as permanent. Did you change it now?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692896660.959199"", ""content"": ""Maybe the reason why we marked it as not required this:\n\n1. When we find a non-permanent flag which has targeting off (just has one variation), we try to see if there's a ticket for it and if not, create one\n2. After creating the ticket, if we go in and add targeting, then the ticket becomes a zombie. \nThere's probably some room for tightening the process.""}, {""user"": ""aganivada"", ""timestamp"": ""1692896735.967369"", ""content"": ""<@U026PMDB1ND> I only turned it on for all tenants. did not make any other changes""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692897043.175859"", ""content"": ""Ok. I guess the reason for marking it as Not Required was because targeting got turned on after creating the ticket. This manual process probably need to be vetted""}]" "1691459904.371619 ","[{""user"": ""aganivada"", ""timestamp"": ""1691459904.371619"", ""content"": ""<@U03KLHDKL1H> regarding is this because it was a new db instance during migration? cc: <@U026PMDB1ND>""}, {""user"": ""askumar"", ""timestamp"": ""1691465162.234169"", ""content"": ""Yes <@U02BV2DGUKC>.. this was later after all the migrations were completed in stage.""}]" "1675070132.783079 ","[{""user"": ""aganivada"", ""timestamp"": ""1675070132.783079"", ""content"": ""<@U02GC8SE18V> <@U03NZ7Z52S2> regarding was this covered in figma? we might be able to fix this in response but just want to know this so we I can update dev-rca appropriately""}, {""user"": ""bganganna"", ""timestamp"": ""1675076432.303169"", ""content"": ""I dont see in figma <@U02BV2DGUKC>""}, {""user"": ""bganganna"", ""timestamp"": ""1675076565.544369"", ""content"": ""<@U02BV2DGUKC> I noticed one more behaviour , If productAdmin removes the product , it gets deleted from the tenant even for orgadmin. Should we delete from tenant or should we remove from that ProductAdmin role ?""}, {""user"": ""aganivada"", ""timestamp"": ""1675076855.687589"", ""content"": ""<@U03NZ7Z52S2> this is expected behaviour once assigned a product admin is admin of the product so he has all permissions as that of orgadmin on the specific product.""}]" "1676879426.013889 ","[{""user"": ""rvaidya"", ""timestamp"": ""1676879426.013889"", ""content"": ""Cross Posting from another thread:\n<@U03ML44ADTQ> <@U033PPLNFRU> <@U04EV0M2VA6>\n\nCan we conclude on this whether we can split the customer table into 2 tables : one for connected customers and another for invited customers?\n\nIf we dont do that, then wrt to pagination,\n1. I will become tricky to support since the 2 set of information (about connected and invited customers) come from different sources. \n2. Then we need to decide if we show the open invites always on top of the connected invites for every page.\nWdyt?\n\ncc <@U02BV2DGUKC> <@U0431DZTPJM> <@U03ML44G5RC> <@U02HCMTQU3W>\n\n> HI <@U03ML44ADTQ>/ <@U033PPLNFRU> , Apps and Platform team are working on pagination of getCustomers page on vendor dashboard and we had a UX related question on this, vendor dashboard landing page is currently listing open invitations where customers have not yet accepted invitation and customers who are already associated with vendor in a single table. So one suggestion that came up during our discussion is if we can separate open invitations from actual customers from ux perspective it will help in pagination related queries since we don\u2019t have to merge and create artificial values for customers where invitations are still open. Also from UI perspective it will look much cleaner if these sections are separated. Please review and check if it makes sense to update ux of this page to separate open invites from actual customers.\n> \n> Here are some discussions on this : \n> \n> \n> cc: <@U02GC8SE18V> <@U02HCMTQU3W>\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1676902037.763239"", ""content"": ""We should split into two tabs to avoid complicating the API implementation and running into issues as a result. Usability might even improve marginally because we can even have specific actions tied to rows in each of the tabs.""}, {""user"": ""rvaidya"", ""timestamp"": ""1676946864.068579"", ""content"": ""<@U03ML44ADTQ> <@U033PPLNFRU> <@U04EV0M2VA6> <@U046ATZ12RL> Can you pls respond ?""}, {""user"": ""avenkataraman"", ""timestamp"": ""1676952009.735739"", ""content"": ""<@U04EV0M2VA6> /<@U046ATZ12RL>: Can you check this?""}, {""user"": ""mli"", ""timestamp"": ""1676952156.716729"", ""content"": ""Hi, apologies for the delay. I\u2019ve been working on some P0 customer demo materials. I\u2019m going to do some quick explorations as suggested by <@U026PMDB1ND> and other folks and will provide an update shortly""}, {""user"": ""mli"", ""timestamp"": ""1676952612.762519"", ""content"": ""<@U02GC8SE18V> does this problem only apply to Customers list on Vendor Dashboard? Or might it also be relevant to other surfaces like Vendor Dashboard > My Team > Teammates and Collaborators, as well as Customer Dashboard > My Team?""}, {""user"": ""apatole"", ""timestamp"": ""1676952638.503849"", ""content"": ""Do we plan on making 2 tabs in other instances where in table there are two different statuses?""}, {""user"": ""rvaidya"", ""timestamp"": ""1676952674.817689"", ""content"": ""<@U04EV0M2VA6> atm, we have this problem only for customers table in vendor dashboard.""}, {""user"": ""apatole"", ""timestamp"": ""1676952930.351899"", ""content"": ""If we are making a change at one instance/page only, then its not uniform UX. Will this problem persist if we provide a default sort like, all the 'invite pending' customers are listed at the end of the list?""}, {""user"": ""aganivada"", ""timestamp"": ""1676953204.081029"", ""content"": ""<@U046ATZ12RL> problem with sorting is if there are too many customers (more than 10) then the invited customers will be in a different page so customer may have to sort multiple times. Also there will not be any scores with invited customers so if we sort by score then these customers will always be at the end of the list.""}, {""user"": ""mli"", ""timestamp"": ""1676954813.954669"", ""content"": ""<@U046ATZ12RL> agree we need to keep UI consistent, but we are exploring adding tabs in List Views in other parts of the app like Pulse List\n\nI lean towards using Tabs instead of sorting and filtering, because:\n\n1. Distinctly displaying categories via tabs will resolve the technical issues listed above \n2. We could treat it as a filter, but our Filters bar can become crowded over time. Also, filters aren\u2019t very visible\n3. With Tabs, we can also adjust the default Columns that are displayed. E.g. for invited customers, \u201coverall score\u201d and \u201cresponse rate\u201d are not applicable\nSee for a design proposal cc <@U02GC8SE18V> <@U026PMDB1ND> <@U02BV2DGUKC>""}]" "1676006557.919879 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1676006557.919879"", ""content"": ""<@U03KLHDKL1H> <@U02BV2DGUKC> - we need to tell Grafana that we cannot do the mapping for individual ids and ideally would like to have a way where we\u2019d periodically dump the mapping into their store and use that in transformations. ""}, {""user"": ""aganivada"", ""timestamp"": ""1676006647.024359"", ""content"": ""Sure <@U026PMDB1ND>""}, {""user"": ""askumar"", ""timestamp"": ""1676006665.324309"", ""content"": ""sure <@U026PMDB1ND>, will update the request""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1676299454.991269"", ""content"": ""<@U03KLHDKL1H>, <@U02BV2DGUKC> - did we hear anything back from Grafana that would simplify this?""}, {""user"": ""askumar"", ""timestamp"": ""1676299595.652739"", ""content"": ""Hi <@U026PMDB1ND> We did get a response as below.\n\n\""\""\""Such mapping is unavailable when using transofrmations. If you need to do it the automated way, then this kind of mapping needs to happen on a datasource level. How do you scrape your metrics? Do you use Prometheus/Grafana Agent or something else? There's a way to relabel your metrics which should do the trick, but the approach (and documentation) varies depending on the software that you're scraping your metrics with.\n\""\""\n\nI have asked for the steps to do it for Prometheus.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1676299753.246019"", ""content"": ""<@U02BV2DGUKC> - the names we need are tenants and products or product-instances?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1676299795.487449"", ""content"": ""In the overall scheme of things, what are the mappings needed?""}, {""user"": ""aganivada"", ""timestamp"": ""1676299980.443959"", ""content"": ""<@U026PMDB1ND> these are the mappings required overall:\n\nMust:\ntenantid -> tenant name\nuser id -> email/user name\n\ngood to have:\npvtid -> product name\nproduct instance -> product name\n\nsince generally customers and vendors are not associated with multiple products not having product instance id to name translation should still be manageable""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1676300251.056419"", ""content"": ""So, if we have a name cache (in Redis) that gets written to (a) whenever an entity gets created or deleted (b) periodically to make sure there are no misses\n\nAnd on the producing side, if this cache is consulted to translate ids to names (with missing translations dropped from metrics) before the metrics are published, would it work?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1676300493.430059"", ""content"": "" is interesting""}, {""user"": ""aganivada"", ""timestamp"": ""1676300674.260799"", ""content"": ""Yes <@U026PMDB1ND> that would work, initially we were planning for it but we also wanted to explore options available from grafana/Prometheus to automatically link it more like a foreign key in db so we don't have to enrich data while we publish metrics cc: <@U0336QZAF98>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1676300795.193529"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1676301115.877469"", ""content"": ""Interesting so are they accessing data directly from SQL? If it is a SQL query then it is ok i guess might not work if grafana had to make a query for each row to resolve the name""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1676309010.310499"", ""content"": ""Maybe this is for on-prem Grafana""}]" "1679570898.093899 ","[{""user"": ""aganivada"", ""timestamp"": ""1679570898.093899"", ""content"": ""<@U03NZ7Z52S2> sanity test suite is taking > 1 hour in stage which is causing the job timeouts. Any recent change that could have caused this?\n\n\n""}, {""user"": ""bganganna"", ""timestamp"": ""1679571652.871319"", ""content"": ""Yes <@U02BV2DGUKC> added delete product and publishers in the tear down but keeping the page size as 100 only .. as we have not deleted the publishers for few tenants from long time it\u2019s taking time ""}, {""user"": ""bganganna"", ""timestamp"": ""1679571690.380339"", ""content"": ""Can we delete the data from the dB?""}, {""user"": ""aganivada"", ""timestamp"": ""1679571715.163059"", ""content"": ""sure Bhavana, if there is a pattern then we can cleanup from DB""}, {""user"": ""aganivada"", ""timestamp"": ""1679571750.310789"", ""content"": ""will this impact prod as well?""}, {""user"": ""bganganna"", ""timestamp"": ""1679573639.333299"", ""content"": ""yes <@U02BV2DGUKC>\n""}, {""user"": ""aganivada"", ""timestamp"": ""1679573803.744179"", ""content"": ""got it thank you <@U03NZ7Z52S2> may be we should cleanup records manually at a time when there isn't much traffic rather than relying on ST to do it ?""}, {""user"": ""bganganna"", ""timestamp"": ""1679573949.916859"", ""content"": ""yes <@U02BV2DGUKC> once we delete the existing records ,from ST we would be cleaning up only fewer records which gets added as part of the test""}, {""user"": ""bganganna"", ""timestamp"": ""1679574019.582469"", ""content"": ""we were only deleting the product instances and pvts, products and publishers were missed as part of teardown""}, {""user"": ""aganivada"", ""timestamp"": ""1679574051.006729"", ""content"": ""let me know when you are free we can first start with stage""}, {""user"": ""aganivada"", ""timestamp"": ""1679581767.767079"", ""content"": ""<@U03NZ7Z52S2> test completed and all 1000+ records in stage cleaned up, triggered a stage automation test now""}, {""user"": ""aganivada"", ""timestamp"": ""1679587839.307729"", ""content"": ""\n\ncc: <@U026PMDB1ND> <@U03NZ7Z52S2>""}, {""user"": ""aganivada"", ""timestamp"": ""1679588061.005879"", ""content"": ""<@U03NZ7Z52S2> please share the git commit link once you update the develop branch""}, {""user"": ""aganivada"", ""timestamp"": ""1679588119.439419"", ""content"": ""<@U026PMDB1ND> this is the cleanup jobs pipeline took ~35 mins to cleanup ~1k records in stage~ 2k records in stage (1k product + 1k publisher)""}, {""user"": ""bganganna"", ""timestamp"": ""1679588445.762839"", ""content"": ""<@U02BV2DGUKC> as we marked the existing testcase to run as part of teardown, test setup and its run is also adding to the execution time.""}, {""user"": ""aganivada"", ""timestamp"": ""1679589271.316629"", ""content"": ""<@U03NZ7Z52S2> but when we ran single test shouldn't setup trigger only once?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679589963.328559"", ""content"": ""Each operation takes about 2 seconds. 1000 ops - 2000 seconds.\n\nEach REST call takes ~400 millis. There are two unnecessary token calls being made which cost 750-800 millis. If we can update the Get Admin S2S Token keyword to just cache the token for the duration of the test, we can reduce time taken for 1000 ops from 2000 to 1250 seconds and be able to delete 1500 records (roughly) in that time.""}, {""user"": ""bganganna"", ""timestamp"": ""1679590361.348349"", ""content"": ""<@U026PMDB1ND> <@U02BV2DGUKC> please review ""}, {""user"": ""aganivada"", ""timestamp"": ""1679591010.208499"", ""content"": ""added a minor comment <@U03NZ7Z52S2>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679598368.927269"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1679749357.844879"", ""content"": ""<@U03NZ7Z52S2> /<@U026PMDB1ND> taking a backup of core prod before triggering cleanup will initially trigger cleanup for 100 records and then issue cleanup of 1000 records in 4 batches""}, {""user"": ""aganivada"", ""timestamp"": ""1679750744.291869"", ""content"": ""\n\ntotal 3994 records switching to deletion of 500 records per batch""}, {""user"": ""aganivada"", ""timestamp"": ""1679760133.653789"", ""content"": ""count is now reduced to 12 and reverted the number of records to be scanned per iteration back to 30\n\n""}]" "1692288973.889709 ","[{""user"": ""aganivada"", ""timestamp"": ""1692288973.889709"", ""content"": ""Very cool demo, and awesome work <@U05D3HNDW3D> :clap::clap:""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1692288995.222179"", ""content"": ""Nice work <@U05D3HNDW3D>""}, {""user"": ""bkathi"", ""timestamp"": ""1692289090.611929"", ""content"": ""thank you!!""}, {""user"": ""bkathi"", ""timestamp"": ""1692289111.071599"", ""content"": ""sorry again for missing the initial slot for platform""}, {""user"": ""aganivada"", ""timestamp"": ""1692289194.303319"", ""content"": ""no problem, I was looking into teams invitee's list didnt find your name so thought missed inviting you""}]" "1685965661.110669 ","[{""user"": ""snangia"", ""timestamp"": ""1685965661.110669"", ""content"": "" Gitlab pipeline in pulse-manager has started failing recently with:\n```Non-resolvable parent POM for com.axm.collaboration:pulse-manager-parent:0.0.19-COLL-2520-SNAPSHOT: Could not transfer artifact com.axm.platform:axm-parent:pom:0.0.22 from/to gitlab-maven (): authentication failed for , status: 401 Unauthorized and 'parent.relativePath' points at no local POM @ line 10, column 11 -> [Help 2]```\n\n""}, {""user"": ""aganivada"", ""timestamp"": ""1685965764.169859"", ""content"": ""<@U04JT69T00K> can you take a look? Please check if clearing runner cache helps""}, {""user"": ""pjha"", ""timestamp"": ""1685965885.197039"", ""content"": ""yes looking into it""}, {""user"": ""pjha"", ""timestamp"": ""1685967106.735419"", ""content"": ""<@U02BV2DGUKC> clearing runner cache didn't help.\nI am able to access when trying manually but in build it's giving 401""}, {""user"": ""aganivada"", ""timestamp"": ""1685970178.534579"", ""content"": ""ok <@U04JT69T00K> all failed builds ran on shared-runner right?""}, {""user"": ""pjha"", ""timestamp"": ""1685970198.482989"", ""content"": ""yes""}, {""user"": ""aganivada"", ""timestamp"": ""1685970246.414779"", ""content"": ""hmmm weird gitlab says everything is operational ""}, {""user"": ""aganivada"", ""timestamp"": ""1685970270.114619"", ""content"": ""ohh ok I know the issue""}, {""user"": ""aganivada"", ""timestamp"": ""1685970290.737589"", ""content"": ""<@U03RSS0S76Y> did we recently upgrade the parent pom?""}, {""user"": ""aganivada"", ""timestamp"": ""1685970645.649019"", ""content"": ""<@U04JT69T00K> this might be the issue ""}, {""user"": ""aganivada"", ""timestamp"": ""1685970677.671229"", ""content"": ""~I just enabled it let me try~ enabled to check project specific access and then disabled ""}, {""user"": ""aganivada"", ""timestamp"": ""1685974068.398669"", ""content"": ""tried explicit access to this project from axm-parent but still getting the same error. Raised a support ticket with gitlab cc: <@U026PMDB1ND> <@U04JT69T00K> ""}, {""user"": ""aganivada"", ""timestamp"": ""1685975171.268909"", ""content"": ""<@U026PMDB1ND> I tried to open a support ticket with gitlab they keep closing the ticket stating \""We have identified that your user record is not associated as a named contact with any organization record, so this ticket is being closed.\""\n\n\nFollowing is the content of the ticket I raised\n```POM download failing while building project with 401 un-authorized\n\nAll builds related to one of our libraries latest pom are failing with the following error\n\n1902 [ERROR] Non-resolvable parent POM for com.axm.collaboration:pulse-manager-parent:0.0.19-SNAPSHOT: Could not transfer artifact com.axm.platform:axm-parent:pom:0.0.22 from/to gitlab-maven (): authentication failed for , status: 401 Unauthorized and 'parent.relativePath' points at no local POM @ line 10, column 11 -> [Help 2]\n\nreference: \n\nThe url is accessible when we visit directly and builds also work fine with previous versions of the library axm-parent.\n\nAll builds are running in shared-runner and we have also disabled \""Allow access to this project with a CI_JOB_TOKEN\"" in axm-parent project.\"" This used to work before here is an instance of a successful run today . Can you kindly check and let us know the root cause for the unauthorized error.```\ncan you kindly check and add a ticket if possible.""}, {""user"": ""aganivada"", ""timestamp"": ""1685975203.703839"", ""content"": """"}, {""user"": ""snangia"", ""timestamp"": ""1685975539.370309"", ""content"": ""no <@U02BV2DGUKC>, pipeline randomly started failing for same pom around 5pm or so: ""}, {""user"": ""aganivada"", ""timestamp"": ""1685975726.998609"", ""content"": ""yes <@U03RSS0S76Y> I tried to mention the same thing in the ticket. But unfortunately they are not allowing to add a support ticket, tried a few options but nothing worked. I see SC builds working fine can you check if this is also using parent 0.0.22? \n\none difference is SC is running on our GL runner whereas PMS is on gitlabs shared-runner""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685976114.217639"", ""content"": ""I\u2019ll follow up. ""}, {""user"": ""aganivada"", ""timestamp"": ""1685976167.659819"", ""content"": ""Thank you <@U026PMDB1ND>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685976203.374129"", ""content"": ""The exact same build was working okay earlier (in term of pom)?""}, {""user"": ""aganivada"", ""timestamp"": ""1685976322.589809"", ""content"": ""Yup PMS always was using 0.0.22 of parent (for almost a release, checked with Hari) builds were successful about 6 hours back""}, {""user"": ""aganivada"", ""timestamp"": ""1685976338.608899"", ""content"": """"}, {""user"": ""aganivada"", ""timestamp"": ""1685976352.896939"", ""content"": """"}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685977386.078929"", ""content"": ""Did we experiment with parent 0.21 (or some other version)?""}, {""user"": ""aganivada"", ""timestamp"": ""1685977588.811479"", ""content"": ""<@U026PMDB1ND>, we tried building core-data on gitlab shared-runner and it succeeded since this was working before we thought there is some issue with integration. I tried to re-build axm-parent master so a new 0.0.22 gets published but that also didn't work.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685978769.004399"", ""content"": ""Okay. I was able to submit a case, so let's see what they come back with.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685978889.517989"", ""content"": ""I'll also ask how to add people to our account so that they can submit cases. That restriction seems idiotic.\n\nJust to double check... you never opened a GitLab case so far?""}, {""user"": ""aganivada"", ""timestamp"": ""1685979083.827119"", ""content"": ""Yes Rama, never opened a gitlab ticket.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685979479.879959"", ""content"": ""Looks like we can manage the allowed submitter list by doing this: \n\nI opened a case with them to create such a project for us.""}, {""user"": ""aganivada"", ""timestamp"": ""1685979653.121369"", ""content"": ""This is weird, I didn't understand the point. Why can't they just check the organisation in email and let us create ticket.""}, {""user"": ""snangia"", ""timestamp"": ""1685979949.596049"", ""content"": ""<@U02BV2DGUKC> <@U026PMDB1ND> pom parent versions are same for both SC and PM""}]" "1687793876.824139 ","[{""user"": ""ppant"", ""timestamp"": ""1687793876.824139"", ""content"": "" Thinking about updating the configs in SSM for multi instance to follow this pattern. The backend code will be more structured in this case. Ref - \n```{\n \""multiInstanceConfigs\"": [\n {\n \""serviceName\"": \""tenancy-service\"",\n \""version\"": \""plat-1\""\n },\n {\n \""serviceName\"": \""core-data-service\"",\n \""version\"": \""plat-1\""\n },\n {\n \""serviceName\"": \""onboarding-app-service\"",\n \""version\"": \""plat-2\""\n }\n ]\n}```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687795340.611099"", ""content"": ""As long as we have a script to\nmanage this and don\u2019t expect people to update it manually this is solid""}, {""user"": ""ppant"", ""timestamp"": ""1687795362.715909"", ""content"": ""Yup, will be done via script :+1:""}]" "1677573370.171269 ","[{""user"": ""aganivada"", ""timestamp"": ""1677573370.171269"", ""content"": "" I have to step out for some personal work, will be away for next 2 hours""}, {""user"": ""aganivada"", ""timestamp"": ""1677581957.149719"", ""content"": ""back now""}]" "1681280961.862829 ","[{""user"": ""araman"", ""timestamp"": ""1681280961.862829"", ""content"": ""Hi <@U02BV2DGUKC> I will not be able to attend today's sprint retro. Sprint will be closed before the retro and new tasks will be added in the backlog by today. 0.9.9 HF on 4/14 will be the major focus for systest for the next 2 days.\nWe will be picking up 0.9.10 items(Giving priority to the ones already available on int for testing) for test planning in next sprint.\ncc <@U03NZ7Z52S2> <@U04RUV6SGU9>""}, {""user"": ""aganivada"", ""timestamp"": ""1681281291.262459"", ""content"": ""sure thank you <@U03DHUAJVMK>""}]" "1686890420.610789 ","[{""user"": ""svummidi"", ""timestamp"": ""1686890420.610789"", ""content"": "" <@U0336QZAF98> - Any of you know how to send Imply Polaris metrics to Lightstep?\nPolaris provides an API to scrape metrics periodically ( every one minute). Below are instructions for integration with Prometheus but we\n""}, {""user"": ""aganivada"", ""timestamp"": ""1686890862.225789"", ""content"": ""<@U040RCBPBEC> we are currently using Amazon open telemetry distro to publish metrics to lightstep. We have a sidecar container for every task publishing the metrics of the specific service to lightstep. May be we can bring up a standalone instance of the distro container to scrap metrics and publish to lightstep.""}, {""user"": ""svummidi"", ""timestamp"": ""1686892464.019229"", ""content"": ""<@U02BV2DGUKC> - A separate instance means, do we need to create another stack or can we add this to existing stack like score-provider?""}, {""user"": ""aganivada"", ""timestamp"": ""1686892701.055529"", ""content"": ""We might be able to add to existing stack as well <@U040RCBPBEC> (unless we want independent tracking) I think it should be a simply config change, currently distro is pointing to localhost address this config is coming from a secret store we could add a new config and make score provider cdk change to read new config.""}, {""user"": ""aganivada"", ""timestamp"": ""1692895791.445209"", ""content"": ""<@U040RCBPBEC> / <@U0336QZAF98> added for this, config might be easy but we need to check what kind of data they publish if it is too much then we might have to filter so we dont run out of space in lightstep. will check from my local and update.""}, {""user"": ""aganivada"", ""timestamp"": ""1693307783.103579"", ""content"": ""<@U040RCBPBEC> These are the metrics we are currently getting from polaris. Can I link this to metrics of score provider in LS?\n\n```# HELP query_count The number of total queries.\n# TYPE query_count gauge\nquery_count{region=\""us-west-2\"",tables=\""[]\"",} 10.0 1693307340000\nquery_count{region=\""us-west-2\"",tables=\""[stage_questions]\"",} 6.0 1693307340000\n# HELP query_time_ms The time, in milliseconds, that it took to complete a query.\n# TYPE query_time_ms summary\nquery_time_ms_count{region=\""us-west-2\"",tables=\""[]\"",} 10.0 1693307340000\nquery_time_ms_sum{region=\""us-west-2\"",tables=\""[]\"",} 115.0 1693307340000\nquery_time_ms{region=\""us-west-2\"",tables=\""[]\"",quantile=\""0.5\"",} 2.0 1693307340000\nquery_time_ms{region=\""us-west-2\"",tables=\""[]\"",quantile=\""0.98\"",} 59.00000762939453 1693307340000\nquery_time_ms{region=\""us-west-2\"",tables=\""[]\"",quantile=\""+Inf\"",} 65.0 1693307340000\nquery_time_ms_count{region=\""us-west-2\"",tables=\""[stage_questions]\"",} 6.0 1693307340000\nquery_time_ms_sum{region=\""us-west-2\"",tables=\""[stage_questions]\"",} 590.0 1693307340000\nquery_time_ms{region=\""us-west-2\"",tables=\""[stage_questions]\"",quantile=\""0.5\"",} 94.0 1693307340000\nquery_time_ms{region=\""us-west-2\"",tables=\""[stage_questions]\"",quantile=\""0.98\"",} 116.05339050292969 1693307340000\nquery_time_ms{region=\""us-west-2\"",tables=\""[stage_questions]\"",quantile=\""+Inf\"",} 124.0 1693307340000```""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1693307820.934229"", ""content"": ""can we have different name instead of score provider to avoid confusion?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1693307837.105399"", ""content"": ""plus do you success and failures count ?""}, {""user"": ""aganivada"", ""timestamp"": ""1693308153.427329"", ""content"": ""<@U0336QZAF98> we need a service to link the metrics since this is a polling model. If we want a separate service then we might have to provision a resource only to poll metrics from Polaris (Lambda or ECS instance etc). Let me check if we can do something while pushing metrics to LS to separate it from rest of score provider metrics.\n\n> plus do you success and failures count ?\nWe should probably get `ingest_events_processed` and `ingest_events_thrownAway` based on their documentation not sure why we dont have these metrics though""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1693308189.334449"", ""content"": ""sure if possible - can you prefix - polaris_ ?""}, {""user"": ""aganivada"", ""timestamp"": ""1693308200.861289"", ""content"": ""sure""}, {""user"": ""aganivada"", ""timestamp"": ""1693308579.184569"", ""content"": ""<@U0336QZAF98> what should be the scrape interval? for regular app metrics we have 60s""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1693308599.662249"", ""content"": ""We can start with 60s for now""}, {""user"": ""aganivada"", ""timestamp"": ""1693308678.944729"", ""content"": ""since this is external service would 60s be a bit chatty? This is however configurable, for int I will start with 60s based on couple of days data may be we can adjust it again""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1693308707.152759"", ""content"": ""An d one more thig is - we use same cluster for all three environmentd""}, {""user"": ""aganivada"", ""timestamp"": ""1693308739.814839"", ""content"": ""we append env flag to metrics before publishing so it should be fine""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1693308762.245819"", ""content"": ""QC ?""}, {""user"": ""aganivada"", ""timestamp"": ""1693308769.681579"", ""content"": ""sure""}, {""user"": ""aganivada"", ""timestamp"": ""1693312500.577509"", ""content"": ""<@U040RCBPBEC> <@U0336QZAF98> here are the metrics from polaris I am working on the changes to publish them from score provider for now have updated experiments service config to publish these metrics. This is just a sample dashboard with all metrics. We can customize the dashboard to show only metrics from int/stage/prod.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1693312524.287659"", ""content"": ""Looks good <@U02BV2DGUKC>""}, {""user"": ""svummidi"", ""timestamp"": ""1693334702.077249"", ""content"": ""<@U02BV2DGUKC> <@U0336QZAF98> - What are the metrics we are collecting? Are we getting any ingestion related also? Can you share list of metrics view?""}, {""user"": ""aganivada"", ""timestamp"": ""1693366753.146189"", ""content"": ""<@U040RCBPBEC> as of now we are not filtering any metrics, is full list of metrics we should get from Polaris . Here are some of the metrics we started to notice.""}, {""user"": ""svummidi"", ""timestamp"": ""1693889952.223889"", ""content"": ""<@U02BV2DGUKC> <@U0336QZAF98> Can we add below metrics to the dashboard. It looks like we need to filter based on the tables using labels to track for each environment. I hope it is possible but I don\u2019t know how to do it in Lightstep. Can we create separate dashboards for each environment?\n\u2022 ingest_job_count\n\u2022 ingest_kafka_lag\n\u2022 ingest_events_processed\n\u2022 ingest_events_thrownAway\n\u2022 ingest_events_unparseable\nWe also need to create alerts on:\n\u2022 ingest_events_unparseable\n\u2022 ingest_kafka_lag\n\u2022 query_time_ms\n""}, {""user"": ""aganivada"", ""timestamp"": ""1693890253.781509"", ""content"": ""<@U040RCBPBEC> we are currently publishing all the metrics that polaris publishes.\n\n> Can we create separate dashboards for each environment?\nSure we can do that\n> We also need to create alerts on:\nsure, I will start a base alert on these metrics and share then we can add thresholds later.""}, {""user"": ""aganivada"", ""timestamp"": ""1693891196.875319"", ""content"": ""<@U040RCBPBEC> /<@U0336QZAF98> we are currently only getting table and query metrics from polaris endpoint do we know if there is any setting in polaris to enable ingestion metrics? I didn't find anything in their documentation \n\n\n```# HELP table_intervals The total number of intervals in the table.\n# TYPE table_intervals gauge\ntable_intervals{region=\""us-west-2\"",table=\""int_metrics\"",} 46.0 1693890780000\ntable_intervals{region=\""us-west-2\"",table=\""int_pulse_instance_state\"",} 8.0 1693890780000\ntable_intervals{region=\""us-west-2\"",table=\""int_questions\"",} 458.0 1693890780000\ntable_intervals{region=\""us-west-2\"",table=\""production_metrics\"",} 15.0 1693890780000\ntable_intervals{region=\""us-west-2\"",table=\""production_pulse_instance_state\"",} 7.0 1693890780000\ntable_intervals{region=\""us-west-2\"",table=\""production_questions\"",} 461.0 1693890780000\ntable_intervals{region=\""us-west-2\"",table=\""stage_metrics\"",} 29.0 1693890780000\ntable_intervals{region=\""us-west-2\"",table=\""stage_pulse_instance_state\"",} 40.0 1693890780000\ntable_intervals{region=\""us-west-2\"",table=\""stage_questions\"",} 464.0 1693890780000\n# HELP query_count The number of total queries.\n# TYPE query_count gauge\nquery_count{region=\""us-west-2\"",tables=\""[]\"",} 26.0 1693890780000\nquery_count{region=\""us-west-2\"",tables=\""[stage_questions]\"",} 38.0 1693890780000\n# HELP table_size The total size of the table in bytes.\n# TYPE table_size gauge\ntable_size{region=\""us-west-2\"",table=\""int_metrics\"",} 189414.0 1693890780000\ntable_size{region=\""us-west-2\"",table=\""int_pulse_instance_state\"",} 19575.0 1693890780000\ntable_size{region=\""us-west-2\"",table=\""int_questions\"",} 9991201.0 1693890780000\ntable_size{region=\""us-west-2\"",table=\""production_metrics\"",} 43515.0 1693890780000\ntable_size{region=\""us-west-2\"",table=\""production_pulse_instance_state\"",} 16016.0 1693890780000\ntable_size{region=\""us-west-2\"",table=\""production_questions\"",} 6031403.0 1693890780000\ntable_size{region=\""us-west-2\"",table=\""stage_metrics\"",} 92806.0 1693890780000\ntable_size{region=\""us-west-2\"",table=\""stage_pulse_instance_state\"",} 93436.0 1693890780000\ntable_size{region=\""us-west-2\"",table=\""stage_questions\"",} 2.4293362E7 1693890780000\n# HELP table_bytes_compacted The total size, in bytes, of the compacted data in the table.\n# TYPE table_bytes_compacted gauge\ntable_bytes_compacted{region=\""us-west-2\"",table=\""int_metrics\"",} 97400.0 1693890780000\ntable_bytes_compacted{region=\""us-west-2\"",table=\""int_pulse_instance_state\"",} 15269.0 1693890780000\ntable_bytes_compacted{region=\""us-west-2\"",table=\""int_questions\"",} 4278611.0 1693890780000\ntable_bytes_compacted{region=\""us-west-2\"",table=\""production_metrics\"",} 11783.0 1693890780000\ntable_bytes_compacted{region=\""us-west-2\"",table=\""production_pulse_instance_state\"",} 2302.0 1693890780000\ntable_bytes_compacted{region=\""us-west-2\"",table=\""production_questions\"",} 1250054.0 1693890780000\ntable_bytes_compacted{region=\""us-west-2\"",table=\""stage_metrics\"",} 49999.0 1693890780000\ntable_bytes_compacted{region=\""us-west-2\"",table=\""stage_pulse_instance_state\"",} 30896.0 1693890780000\ntable_bytes_compacted{region=\""us-west-2\"",table=\""stage_questions\"",} 4494727.0 1693890780000\n# HELP table_intervals_compacted The number of intervals in the table that have been compacted.\n# TYPE table_intervals_compacted gauge\ntable_intervals_compacted{region=\""us-west-2\"",table=\""int_metrics\"",} 26.0 1693890780000\ntable_intervals_compacted{region=\""us-west-2\"",table=\""int_pulse_instance_state\"",} 6.0 1693890780000\ntable_intervals_compacted{region=\""us-west-2\"",table=\""int_questions\"",} 128.0 1693890780000\ntable_intervals_compacted{region=\""us-west-2\"",table=\""production_metrics\"",} 4.0 1693890780000\ntable_intervals_compacted{region=\""us-west-2\"",table=\""production_pulse_instance_state\"",} 1.0 1693890780000\ntable_intervals_compacted{region=\""us-west-2\"",table=\""production_questions\"",} 91.0 1693890780000\ntable_intervals_compacted{region=\""us-west-2\"",table=\""stage_metrics\"",} 16.0 1693890780000\ntable_intervals_compacted{region=\""us-west-2\"",table=\""stage_pulse_instance_state\"",} 15.0 1693890780000\ntable_intervals_compacted{region=\""us-west-2\"",table=\""stage_questions\"",} 109.0 1693890780000\n# HELP query_time_ms The time, in milliseconds, that it took to complete a query.\n# TYPE query_time_ms summary\nquery_time_ms_count{region=\""us-west-2\"",tables=\""[]\"",} 26.0 1693890780000\nquery_time_ms_sum{region=\""us-west-2\"",tables=\""[]\"",} 177.0 1693890780000\nquery_time_ms{region=\""us-west-2\"",tables=\""[]\"",quantile=\""0.5\"",} 1.769981026649475 1693890780000\nquery_time_ms{region=\""us-west-2\"",tables=\""[]\"",quantile=\""0.98\"",} 66.6399917602539 1693890780000\nquery_time_ms{region=\""us-west-2\"",tables=\""[]\"",quantile=\""+Inf\"",} 76.0 1693890780000\nquery_time_ms_count{region=\""us-west-2\"",tables=\""[stage_questions]\"",} 38.0 1693890780000\nquery_time_ms_sum{region=\""us-west-2\"",tables=\""[stage_questions]\"",} 7219.0 1693890780000\nquery_time_ms{region=\""us-west-2\"",tables=\""[stage_questions]\"",quantile=\""0.5\"",} 211.0 1693890780000\nquery_time_ms{region=\""us-west-2\"",tables=\""[stage_questions]\"",quantile=\""0.98\"",} 376.800048828125 1693890780000\nquery_time_ms{region=\""us-west-2\"",tables=\""[stage_questions]\"",quantile=\""+Inf\"",} 392.0 1693890780000```""}, {""user"": ""svummidi"", ""timestamp"": ""1693892556.939299"", ""content"": ""I created a support ticket with Polaris\n""}, {""user"": ""aganivada"", ""timestamp"": ""1693892580.249869"", ""content"": ""Thank you <@U040RCBPBEC>""}, {""user"": ""aganivada"", ""timestamp"": ""1693892874.069589"", ""content"": ""<@U040RCBPBEC> env specific dashboards => once we have ingestion metrics we can add to the dashboards""}, {""user"": ""bot_message"", ""timestamp"": ""1693893103.690779"", ""content"": ""@anil created a Task PLAT-2518 Polaris metrics - dashboards & alerts""}]" "1683657373.837949 ","[{""user"": ""hchintamreddy"", ""timestamp"": ""1683657373.837949"", ""content"": "" I was looking to add the following to com.axm.platform.commons.config.WebClientConfig#s2sWebClient so we can figure out which service the call originated from , currently this defaults to \u201cReactorNetty/1.0.22\u201d, let me know what you think\n\n.defaultHeader(\u201cUser-Agent\u201d*,* System._getenv_(\u201cSERVICE_NAME\u201d))""}, {""user"": ""svummidi"", ""timestamp"": ""1683658652.253159"", ""content"": ""Looks like a good idea, if we can prefix with something like \u201cInternalService-\u201c, we can use it for filtering with common prefix.""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1683662637.333559"", ""content"": ""sure <@U040RCBPBEC> will add it with the prefix""}, {""user"": ""aganivada"", ""timestamp"": ""1683689235.940149"", ""content"": ""sure thank you <@U02D4DUKDQC>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683733347.149509"", ""content"": ""<@U02D4DUKDQC> - might make sense to shorten the prefix. Could use \""axm-\"" for instance so that the final value would look like axm-message-broker, axm-onboarding-app etc.""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1683744574.757189"", ""content"": ""sure <@U026PMDB1ND> will fix this""}]" "1688678414.314289 ","[{""user"": ""svummidi"", ""timestamp"": ""1688678414.314289"", ""content"": "" - Some logs showing with message and some with message_obj.msg - Is there any way to fix this?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1688698379.747259"", ""content"": ""<@U040RCBPBEC> - I think this depends on component which is logging this statement - the one you see in message - are mostly spring default ones - where as one you see in message_obj.msg is typically ours - and typically they will have extra context under key message_obj.*""}, {""user"": ""aganivada"", ""timestamp"": ""1688712127.222339"", ""content"": ""<@U040RCBPBEC> <@U0336QZAF98> this happens based on the content in logs if log contains json payload it will get transformed into message_obj \n\ninitially when we started with logz we had an issue where if we had json payload in the body they'd just turn it a one big string with quotes instead of parsing into individual records so logz added this pipeline to transform object wherever it is required. cc: <@U026PMDB1ND>""}]" "1681268605.858639 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1681268605.858639"", ""content"": "" - here is the product metrics spec from Alex, in case you have not seen it yet - ""}, {""user"": ""askumar"", ""timestamp"": ""1681268864.715439"", ""content"": ""Thanks <@U026PMDB1ND>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681268965.317829"", ""content"": ""One option for us to build this is out is to\n(a) build/repurpose some admin APIs to fetch data from our services\n(b) push the metrics that we are now pushing to Grafana to Snowflake\n(c) Surface both in a Retool app.\n\nI see that Retool's charting is not advanced but this way we can get the basic views in place.\n\nIf advanced charting UI is needed we need to find something only for visualizing Snowflake data but we'd still need some way to surface the data in the DB.""}, {""user"": ""askumar"", ""timestamp"": ""1681269178.351749"", ""content"": ""<@U026PMDB1ND> We are currently exploring the option (b) , where we are pushing the metric using an event based model. \nWe push it to lamba and lambda pushes it to appropriate table in snowflake based on event type identifier.\ncc <@U02BV2DGUKC>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681269264.932419"", ""content"": ""How many tables did we define in Snowflake?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681269377.536539"", ""content"": ""SF does very well with Json data so we could potentially put a bunch of events in the same table and use type to differentiate them. I don't think that the way we define tables in RDBMS maps 1:1 to Snowflake.""}, {""user"": ""askumar"", ""timestamp"": ""1681269479.255239"", ""content"": ""Sure <@U026PMDB1ND>...one to one mapping can lead to piling up of tables.\nFor demo we created a sample test scheme with 3 tables.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681269479.418199"", ""content"": ""Don't plan on Joins and such. They do extremely well with compression, So even if the events are verbose it is okay""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681269508.055549"", ""content"": ""Don't plan on Joins and such. ==> Don't plan on too many joins""}, {""user"": ""askumar"", ""timestamp"": ""1681269597.359139"", ""content"": ""Ack <@U026PMDB1ND> will try to take care of this for our new schema.\ncc <@U02BV2DGUKC> <@U02SF36PVKL>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681269733.294949"", ""content"": ""If we look at all the events we are generating and if we can come up with a \""union\"" schema, we might be able to get a lot of mileage.""}, {""user"": ""aganivada"", ""timestamp"": ""1681269842.578809"", ""content"": ""sure <@U026PMDB1ND>, for some cases we may need joins like tenant id to name and user id to name. It can be handled without join also but it might be a bit constly because we'd have to trigger id to name for every event""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681313696.080109"", ""content"": ""<@U02BV2DGUKC> - user and tenant can be in their tables since they have a structure very different from any event. I meant to say that we should see if we can avoid creating a table per event type.""}, {""user"": ""aganivada"", ""timestamp"": ""1681313810.632369"", ""content"": ""> user and tenant can be in their tables\nsure Rama, these can be snowflake tables that are also updated by events right or should we enrich data before publishing the event? we were thinking if data is in snowflake tables then we can avoid making multiple calls to convert id to name""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681321534.606609"", ""content"": ""I think we can leave them as ids. Let's see how things work in that mode.""}]" "1684215296.814089 ","[{""user"": ""aganivada"", ""timestamp"": ""1684215296.814089"", ""content"": ""<@U02SCRTM2M7> <@U0431DZTPJM> can we get on a call to look into the report-generator lambda issue?""}, {""user"": ""aganivada"", ""timestamp"": ""1684216490.950119"", ""content"": ""issue is resolved now, sorry I wasn't aware that we didn't have the report generation private dns changes merged to develop yet and I deployed report-generator from develop for testing cdkv2""}]" "1678821539.394609 ","[{""user"": ""mnirmal"", ""timestamp"": ""1678821539.394609"", ""content"": "" I have been seeing this issue - 0.0.14 version of com.axm.commons\n```ERROR Unable to invoke factory method in class com.axm.platform.logging.util.AxmFilter for element AxmFilter: java.lang.NullPointerException java.lang.reflect.InvocationTargetException\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)```\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678821624.461939"", ""content"": ""<@U02SF36PVKL> is there a cause trace as well?\n\nOr is this the whole trace?""}, {""user"": ""mnirmal"", ""timestamp"": ""1678821658.833099"", ""content"": ""fyi, this is happening when I run the service locally.\n```2023-03-15 00:44:10,473 main ERROR Null object returned for AxmFilter in Filters.\n00:44:10.639 [main] INFO [{}] o.sp.te.con.support.AbstractContextLoader - Could not detect default resource locations for test class [com.axm.collaboration.sentiment.collector.SlackAddressBookConstructorTest]: no resource found for suffixes {-context.xml, Context.groovy}.\n00:44:10.978 [main] INFO [{}] o.sp.bo.tes.context.SpringBootTestContextBootstrapper - Loaded default TestExecutionListener class names from location [META-INF/spring.factories]: [org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener, org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListener, org.springframework.boot.test.autoconfigure.restdocs.RestDocsTestExecutionListener, org.springframework.boot.test.autoconfigure.web.client.MockRestServiceServerResetTestExecutionListener, org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrintOnlyOnFailureTestExecutionListener, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverTestExecutionListener, org.springframework.boot.test.autoconfigure.webservices.client.MockWebServiceServerTestExecutionListener, org.springframework.test.context.web.ServletTestExecutionListener, org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener, org.springframework.test.context.event.ApplicationEventsTestExecutionListener, org.springframework.test.context.support.DependencyInjectionTestExecutionListener, org.springframework.test.context.support.DirtiesContextTestExecutionListener, org.springframework.test.context.transaction.TransactionalTestExecutionListener, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener, org.springframework.test.context.event.EventPublishingTestExecutionListener, org.springframework.security.test.context.support.WithSecurityContextTestExecutionListener, org.springframework.security.test.context.support.ReactorContextTestExecutionListener]\n00:44:11.011 [main] INFO [{}] o.sp.bo.tes.context.SpringBootTestContextBootstrapper - Using TestExecutionListeners: [org.springframework.test.context.web.ServletTestExecutionListener@592238c5, org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener@6ee8dcd3, org.springframework.test.context.event.ApplicationEventsTestExecutionListener@a20b94b, org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener@5cdf39b2, org.springframework.boot.test.autoconfigure.SpringBootDependencyInjectionTestExecutionListener@14f3c6fc, org.springframework.test.context.support.DirtiesContextTestExecutionListener@3cd59ef5, org.springframework.test.context.transaction.TransactionalTestExecutionListener@33b082c5, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener@16073fa8, org.springframework.test.context.event.EventPublishingTestExecutionListener@1cc9cfb2, org.springframework.security.test.context.support.WithSecurityContextTestExecutionListener@cfbc8e8, org.springframework.security.test.context.support.ReactorContextTestExecutionListener@3bead518, org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListener@14bf57b2, org.springframework.boot.test.autoconfigure.restdocs.RestDocsTestExecutionListener@46d9aec8, org.springframework.boot.test.autoconfigure.web.client.MockRestServiceServerResetTestExecutionListener@14379273, org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrintOnlyOnFailureTestExecutionListener@7918c7f8, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverTestExecutionListener@17740dae, org.springframework.boot.test.autoconfigure.webservices.client.MockWebServiceServerTestExecutionListener@1c504e66]\n2023-03-15 00:44:11,714 main ERROR Unable to invoke factory method in class com.axm.platform.logging.util.AxmFilter for element AxmFilter: java.lang.NullPointerException java.lang.reflect.InvocationTargetException\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\t```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678821820.372239"", ""content"": ""Is this is the first time you included commons 0.0.14?""}, {""user"": ""mnirmal"", ""timestamp"": ""1678821851.144229"", ""content"": ""No, i didn't change any versions. Just noticed this""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678821917.796059"", ""content"": ""Does this prevent your test from starting?""}, {""user"": ""mnirmal"", ""timestamp"": ""1678821953.570759"", ""content"": ""No, the test runs fine. But I see these errors.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678821964.985879"", ""content"": ""Oh okay""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678822604.526479"", ""content"": ""<@U02SF36PVKL> - can you make a small change in test/resources/log4j2.xml? Update `<AxmFilter maxApplicationLevel=\""info\""/>` to\n`<AxmFilter maxApplicationLevel=\""info\"" tenants=\""75898\""/>`\n\nJust to verify a theory quickly. Please go back to the default state after that.""}, {""user"": ""mnirmal"", ""timestamp"": ""1678823375.116769"", ""content"": ""<@U026PMDB1ND> the above change fixed it thanks.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678823638.085939"", ""content"": ""Cool, thanks <@U02SF36PVKL>. I think we need to fix a bit of code in observability to truly fix it.\n\n<@U0431DZTPJM>, I guess\n`public static TenantsLogConfig convert(String jsonString) {`\nneeds a null check at the top.\n\nLooks like we are getting a null but not a \""\"" when nothing is specified for tenants.\n `@PluginAttribute(value = \""tenants\"", defaultString = \""\"") final String listOfTenants,`\n\n\nfyi <@U02BV2DGUKC>""}]" "1681346510.959149 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1681346510.959149"", ""content"": "" - we have a requirement to build product metrics dashboards. Basically, the data could come from Salesforce, bunch of Axm tables, maybe the audit-log ES store and some event data we'd push to Snowflake\n\nHere are some of the initial set of dashboards that PM is asking for: \n\nHere are some of our options for building the middleware/backend APIs to drive these dashboards (which we would build using Retool instead of building our own UI):\n\n:one: Build an admin app middleware service (using reactive Java) and add all the required APIs to the backend services.\n Pros: Same stack, same as everything else we do.\n Cons: Releases would be heavy. Platform team should either ask for backend APIs or make additions in multiple services and maintain those changes(!)\n\n\n:two: Build an admin service in Python (to develop flexible & fast APIs). Allow it connect to all the data stores but only execute SELECT queries and return the data required. Can't force the use of reactive Java as blocking DB calls would be a challenge and RDB is limiting.\n Pros: No dependence on backend teams to deliver the required functionality.\n Cons: Different stack; Data access in the databases would be from outside the service that owns the schema, though only SELECTs would ensure that all backward compatible changes would be okay.\n\n\n:three: Same as 2, but instead of building a service build a set of lambdas and keep things even looser.\n Pros: Lot easier to develop and deploy than a server type of a thing. Releasing is trivial. Maybe cost-effective considering the sparse load.\n Cons: Could lead to some duplication. Caching anything would be harder.\n\n:four: Build (2) but in non-reactive Java instead of Python. Use JPA or plain JDBC to connect to the data stores.\n\n\nWhich of these options sounds good to you? React with the choice. Are there others you can think of?\n\ncc <@U02BV2DGUKC>""}, {""user"": ""svummidi"", ""timestamp"": ""1681350035.374939"", ""content"": ""Based on Pros and Cons - I feel something like #2 is better. It need not be Python but anything not as heavy as Java because it is for our internal consumption.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681350267.421729"", ""content"": ""One advantage with Java is that the rest of the stack (observability, ff, etc.) are free. Agree that it is a bit heavy.\n\nI think that we should either do Java or Python because another language would make it too specialized for people to add more things (basically there's no technical reason to not try NodeJS w/ Typescript for example, but it would be too specialized).\n\nLet's see what others say as well :slightly_smiling_face:""}, {""user"": ""ppant"", ""timestamp"": ""1681351105.009539"", ""content"": ""I think #2 seems better because, like Satya said, it won\u2019t be as heavy as Java. Also if there comes a scenario of building APIs that consolidate many data sources, we can plan to go GraphQL route (Python has a good library called Graphene for this)""}, {""user"": ""aganivada"", ""timestamp"": ""1681353499.278639"", ""content"": ""<@U026PMDB1ND> one question on (#2) since we are using retool if a backend service already has an endpoint we are looking for should we replicate the endpoint in python or have retool connect directly to backend via s2s?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681353645.330129"", ""content"": ""I don\u2019t think we\u2019d have too many cases where the existing APIs are a perfect fit, <@U02BV2DGUKC> ""}, {""user"": ""akasim"", ""timestamp"": ""1681355556.286939"", ""content"": ""<@U026PMDB1ND> In case of :two: how we are handling the data from third-party? \nFor Salesforce, are we expecting another backend service to manage this data? or the Python middleware will poll it from the customer endpoint? And where do we persist in that case ?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1681359154.254839"", ""content"": ""<@U026PMDB1ND> - Retool and other tools support connecting via jdbc directly - whether it is rds or snowflake . So unless we have an issue - where connecting to jdbc is not an option (bcoz of vpc or network) - we can go with option 2) - else connect via jdbc directly to datasources instead of having another wrapper - required views for retool can be created in db layer itself .""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681359846.273829"", ""content"": ""> For Salesforce, are we expecting another backend service to manage this data? or the Python middleware will poll it from the customer endpoint? And where do we persist in that case\n<@U02HQ78V9A5> We are not getting data from a customer's Salesforce. It would be from Axiamatic's SF. We can try to connect Retool to SF (pereferably) or add some code to the middleware service to talk to SF""}, {""user"": ""akasim"", ""timestamp"": ""1681359932.356019"", ""content"": ""Ah, got it.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681360014.726239"", ""content"": ""> else connect via jdbc directly to datasources instead of having another wrapper\n<@U0336QZAF98> - it is not likely that we'd just take data from one table and paint a grid, so this direct JDBC connectivity is not going to be sufficient. Also, when enterprise customers want to check out security perimeter it would be easier to convince them of our posture if it doesn't encompass Retool.\n\nSo we can't get away without some middleware""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681360138.245459"", ""content"": ""For example, we'd have to talk to tenancy for customer names, PM for some part of the data and maybe SC for some other part of the data. Essentially, we'd be doing what apps is doing now.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1681360228.261349"", ""content"": ""Why not have Java 21 w/ light threads instead of Python? :wink:""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1681360419.949529"", ""content"": ""Agreed <@U026PMDB1ND> from security posture perspective .But purpose of retool is stitching data across multiple services - which we are trying to do in wrapper - and people will always have different requirement or views - at that time we dont want to compile""}, {""user"": ""aganivada"", ""timestamp"": ""1681365110.119929"", ""content"": ""> Why not have Java 21 w/ light threads instead of Python? :wink:\n> \nLove the idea :grin:""}]" "1693464104.836679 ","[{""user"": ""akasim"", ""timestamp"": ""1693464104.836679"", ""content"": ""\nGitlab pipeline is again going to pending state. The pipeline which was triggered 15mins back is still pending. Would be great if someone can help here, as we need to rollout the HF today.\ncc: <@U02BV2DGUKC>""}, {""user"": ""akasim"", ""timestamp"": ""1693464124.679099"", ""content"": """"}, {""user"": ""akasim"", ""timestamp"": ""1693464134.448969"", ""content"": ""cc: <@U04JT69T00K>""}, {""user"": ""pjha"", ""timestamp"": ""1693464169.254619"", ""content"": ""<@U02HQ78V9A5> checking it""}, {""user"": ""pjha"", ""timestamp"": ""1693465016.087199"", ""content"": ""<@U02HQ78V9A5> pipeline started, still looking into the root cause""}, {""user"": ""akasim"", ""timestamp"": ""1693465026.081879"", ""content"": ""Sure thanks <@U04JT69T00K>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1693501431.709219"", ""content"": ""<@U04JT69T00K> - what was the fix? Can you please document it under HowTO""}]" "1685527350.914489 ","[{""user"": ""ppant"", ""timestamp"": ""1685527350.914489"", ""content"": "" I have updated the doc for backend changes for multi instances , will be adding more details and links to infra docs as well. Please review this and let me know if some changes are required""}, {""user"": ""ppant"", ""timestamp"": ""1685579375.005169"", ""content"": "" Just a reminder""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685579563.467199"", ""content"": ""<@U0431DZTPJM> - this looks good in general, but we should call a review meeting with the overall team and walk them through this.\n\nFri AM IST could be an option.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685579634.419749"", ""content"": ""The rewiring of Clients (one of the reviews you submitted) makes me wonder if this requires widespread changes everywhere.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685579659.102489"", ""content"": ""We need to have clarity on that before going into the meeting with everyone""}, {""user"": ""ppant"", ""timestamp"": ""1685579700.449669"", ""content"": ""<@U026PMDB1ND> Yesterday I looked through some data and collab services. They were only creating a single client. Will confirm this for all spring services today""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685579843.951269"", ""content"": ""Core data might be forcing a ton of clients to be created in Apps. That could force a major change.""}, {""user"": ""ppant"", ""timestamp"": ""1685580693.226389"", ""content"": ""No Rama. Core-data creates different APIs and Delegates but it only creates a common APIClient. In pom.xml of specification for all the webclient generation for core-data, the `invokerPackage` tag (which defines the package for APIClient) is same""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685589129.318919"", ""content"": ""Phew!!!""}, {""user"": ""ppant"", ""timestamp"": ""1685592137.883189"", ""content"": ""<@U026PMDB1ND> These are the places where clients are generated in our services. All of them are using a single client (except v2)""}, {""user"": ""aganivada"", ""timestamp"": ""1685631537.958849"", ""content"": ""<@U0431DZTPJM> / <@U04JT69T00K> Can we schedule a review meeting Friday morning IST or Monday evening IST ?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1685641466.647579"", ""content"": ""Did we set one up?""}, {""user"": ""aganivada"", ""timestamp"": ""1685677865.897629"", ""content"": ""no <@U026PMDB1ND>, will check with Prabhu and Prashant most likely early next week""}, {""user"": ""ppant"", ""timestamp"": ""1685677906.198089"", ""content"": ""<@U026PMDB1ND> <@U02BV2DGUKC> I did not set it up for today because of the SSO prep call. Will do early next week""}, {""user"": ""aganivada"", ""timestamp"": ""1686047340.266179"", ""content"": ""<@U0431DZTPJM> when are we planning for review meeting?""}]" "1677556536.414679 ","[{""user"": ""aganivada"", ""timestamp"": ""1677556536.414679"", ""content"": ""re-posting from 1-1 chat\n\nwhat would be the value of ${env_role_arn} in run-robot-cicd.sh script of system tests.\nSeems like it's using some value stored in AGENT_ROLE_ARN_INT gitlab variable?\n```function dyn_var() {\n var_name=\""$1\""_\""$2\""\n echo ${var_name}\n val=${!var_name}\n echo ${val}\n if [ \""$val\"" != \""\"" ]; then\n echo $val\n return 0\n else\n var_name=\""$1\""\n val=${!var_name}\n if [ \""$val\"" != \""\"" ]; then\n echo $val\n return 0\n fi\n fi\n echo \""\""\n return 1\n}\n\nAXM_ENV=$(echo \""$AXM_ENV\"" | tr '[:lower:]' '[:upper:]')\nAXM_ENV_LOWER=$(echo \""$AXM_ENV\"" | tr '[:upper:]' '[:lower:]')\n\n\nenv_role_arn=$(dyn_var AGENT_ROLE_ARN $AXM_ENV)```\n(edited)\n\n=> we have these variables for each env for int value is arn:aws:iam::433798924509:role/Gitlab-Agent-Role , actual values come from gitlab variables""}, {""user"": ""snangia"", ""timestamp"": ""1677559541.505279"", ""content"": ""seems the role `arn:aws:iam::433798924509:role/Gitlab-Agent-Role` only works from system test repo and not any other repo, fails with\n```$ STS_RESPONSE=($(aws sts assume-role-with-web-identity \\ # collapsed multi-line command\nAn error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity```\nlooks like there's some further repo level filtering added on aws side for the roles.\n""}, {""user"": ""aganivada"", ""timestamp"": ""1677559755.975749"", ""content"": ""possibly some additional env-var is being passed to system-test job. <@U04JT69T00K> can you try taking a look at this?""}, {""user"": ""pjha"", ""timestamp"": ""1677559886.656289"", ""content"": ""Sure, I will look into it""}, {""user"": ""snangia"", ""timestamp"": ""1677559958.237559"", ""content"": """"}, {""user"": ""snangia"", ""timestamp"": ""1677560190.656079"", ""content"": """"}, {""user"": ""snangia"", ""timestamp"": ""1677560204.565279"", ""content"": ""it's part of Trust Relationships in role""}, {""user"": ""aganivada"", ""timestamp"": ""1677560242.856009"", ""content"": ""cool thank you <@U03RSS0S76Y>""}, {""user"": ""snangia"", ""timestamp"": ""1677560247.080899"", ""content"": ""<@U04JT69T00K> I will let you know the repo name, please add it to this""}, {""user"": ""pjha"", ""timestamp"": ""1677561324.385249"", ""content"": ""ok""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677599742.682139"", ""content"": ""<@U02BV2DGUKC> - we are all good on this topic, right?""}, {""user"": ""aganivada"", ""timestamp"": ""1677599781.837419"", ""content"": ""yes <@U026PMDB1ND>""}]" "1690581213.247069 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1690581213.247069"", ""content"": ""Hey <@U02HQ78V9A5> - looks like you inadvertently reverted the changes made by multiple folks in cdk-artifacts. Maybe you thought you were in a feature branch, but ended up updating develop?\n\n<@U040RCBPBEC> found that the askai changes were gone, but when we probed a bit more, found that this commit undid a bunch of files.\n\nI undid your undo by doing \""git revert aa708a28f56c95b3903518fd22cc8600c178bdf3\"". There was a conflict in core/cdk.json in the \""s3-notifications\"" area. Please check if things are back as they need to be.\n\n - we need to see the various merges we made between Jul 26 and now are available or if we need to replay each of the merges again (list is below).\n\nThe challenge seems to be with branches that were created after Jul 26 and merged back. <@U02BV2DGUKC> <@U03KLHDKL1H> please check that your actual commit content is in place.\n\nMost of the changes seem to be in plat and data. Only one change is in collab.\n\n```git show --stat 892cfe640a84e26eaa5bee3f50df73e8f30aca97\ncommit 892cfe640a84e26eaa5bee3f50df73e8f30aca97\nMerge: c6019417 b943a048\nAuthor: anzar kasim <akasim@axiamatic.com>\nDate: Wed Jul 26 15:57:51 2023 +0530\n\n Merge remote-tracking branch 'origin/develop' into develop\n\n apps/askai-service/askai/askai_base_stack.py | 33 +++++++++++++++++++++++++++------\n apps/askai-service/askai/askai_constants.py | 6 ++++++\n apps/askai-service/cdk.json | 8 ++++++--\n apps/audit-log-lambda/cdk.json | 3 ++-\n apps/config-service/cdk.json | 8 +++++++-\n apps/config-service/config_server/config_service_constants.py | 3 ++-\n apps/config-service/config_server/config_service_stack.py | 13 +++++++++++++\n apps/coordination-building-blocks/cdk.json | 3 ++-\n apps/coordination-building-blocks/coordination_building_blocks/event_handler_lambda_stack.py | 4 ----\n apps/coordination-building-blocks/state-machines/tenancy-tenant-deletion.json | 8 ++++++++\n apps/estimation-service/cdk.json | 3 ++-\n apps/ingestion-service/cdk.json | 4 +++-\n apps/metrics-manager-service/metrics_manager/metrics_manager_stack.py | 12 ++++++++++--\n automated-tests-v1.smod | 1 +\n bootstrap/setup-basics.sh | 20 ++++++++++++++++++++\n bootstrap/setup-functions.sh | 1 +\n core/cdk.json | 29 ++++++++++++++++++++++++++++-\n core/cdk_app.py | 1 +\n core/commons/constants/service_constants.py | 7 +++++++\n core/stacks/s3_notifications_stack.py | 21 +++++++++++++++++++++\n 20 files changed, 167 insertions(+), 21 deletions(-)```""}, {""user"": ""askumar"", ""timestamp"": ""1690622102.961129"", ""content"": ""Thankyou <@U026PMDB1ND> my commit content is there in develop.""}, {""user"": ""akasim"", ""timestamp"": ""1690679038.983149"", ""content"": ""Apologies team.\n<@U026PMDB1ND> I had reverted the merge commit which I had done accidentally from the develop branch.:grimacing: \nThanks for fixing it.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1690692675.604379"", ""content"": ""no problem, <@U02HQ78V9A5>. I think Satya lost more time than anyone else :slightly_smiling_face:""}, {""user"": ""aganivada"", ""timestamp"": ""1690779486.023039"", ""content"": ""thank you <@U026PMDB1ND> , I see my changes as well in develop branch.""}]" "1692834225.323139 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1692834225.323139"", ""content"": "" - In a code review, <@U03KLHDKL1H> asked a question about what I am using for Python code formatting as the diff changed a few lines.\n\nJust wanted to share what I am doing so that at the end of this thread hopefully we can come up with something that works for all.\n\n1. I am using VS Code. \n2. I've installed Black and Flake8 extensions (Microsoft 'authored')\n3. I have the following settings (Preferences -> Settings and the second button from the right (see below) and have the following contents:\n```\""[python]\"": {\n \""gitlens.codeLens.symbolScopes\"": [\n \""!Module\""\n ],\n \""editor.defaultFormatter\"": \""ms-python.black-formatter\"",\n \""editor.formatOnType\"": true,\n \""editor.formatOnSave\"": true,\n \""editor.formatOnPaste\"": true,\n },\n\""black-formatter.args\"": [\""--line-length\"", \""120\""],\n\""editor.defaultFormatter\"": \""ms-python.black-formatter\"",```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692834456.579289"", ""content"": """"}, {""user"": ""rsrinivasan"", ""timestamp"": ""1692842518.218079"", ""content"": ""For pycharm folks - Setup pre-commit hooks in your git repository - which typically contain the same rules <@U026PMDB1ND> has shared. So before commit - it gets formatted .""}, {""user"": ""askumar"", ""timestamp"": ""1692846986.103219"", ""content"": ""Thankyou <@U026PMDB1ND>""}]" "1690905827.239689 ","[{""user"": ""pjha"", ""timestamp"": ""1690905827.239689"", ""content"": "" Please review how to for temporary bastion access\n""}, {""user"": ""askumar"", ""timestamp"": ""1690991014.580309"", ""content"": ""<@U04JT69T00K> Is there a time based clean up of this , does it cleanup the uploaded key automatically after?""}, {""user"": ""pjha"", ""timestamp"": ""1691125964.424829"", ""content"": ""<@U03KLHDKL1H> No, there is no time based clean up""}, {""user"": ""pjha"", ""timestamp"": ""1691126007.353379"", ""content"": ""we manually have to remove keys from the s3 bucket .""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1691130003.125859"", ""content"": ""<@U04JT69T00K> Why can't we do the clean up just like the other thing you implemented where a lambda removes the multi-instance node? Not urgent, but is this doable?""}, {""user"": ""pjha"", ""timestamp"": ""1691131482.327989"", ""content"": ""yes, <@U026PMDB1ND> it is doable. I will work on it.""}]" "1673348529.785699 ","[{""user"": ""araman"", ""timestamp"": ""1673348529.785699"", ""content"": ""Its a test tenant in prod that I own <@U03KLHDKL1H>""}, {""user"": ""askumar"", ""timestamp"": ""1673348584.693229"", ""content"": ""Are you facing any issue with this?, there was some rule parsing error :\nmsg\"": \""rule parsing failed based on splits count = 14037::customer::14037::productInstance::9408::readWrite\""""}, {""user"": ""araman"", ""timestamp"": ""1673348752.166229"", ""content"": ""<@U03NZ7Z52S2> Is this related to the latest update in prod for reports?""}, {""user"": ""bganganna"", ""timestamp"": ""1673348870.707469"", ""content"": ""not sure , reports downloading was timing out as lambda was not triggered <@U02SCRTM2M7> is this err could be becoz of that""}, {""user"": ""sfarooqh"", ""timestamp"": ""1673348994.684939"", ""content"": ""<@U03NZ7Z52S2> does not look like, the lambda timeout was a different issue related to EB rule deployment""}, {""user"": ""sfarooqh"", ""timestamp"": ""1673349018.439129"", ""content"": ""<@U03KLHDKL1H> can you share the trace_id?""}, {""user"": ""askumar"", ""timestamp"": ""1673349037.039009"", ""content"": ""63bd34849ceeb816b14cb6d8f524b705""}, {""user"": ""askumar"", ""timestamp"": ""1673351904.460859"", ""content"": ""Created a tracker for this : \nPTAL and prioritise if required urgently""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1673357981.352239"", ""content"": ""<@U03DHUAJVMK> <@U03NZ7Z52S2> - could you confirm - if there was any manual editing of user metdata - done for this user in auth0""}, {""user"": ""bganganna"", ""timestamp"": ""1673358072.200029"", ""content"": ""No <@U0336QZAF98> not for anu's tenant, but for automation tenant we are calling delete permission api from ST""}, {""user"": ""sfarooqh"", ""timestamp"": ""1673358192.929769"", ""content"": ""<@U0336QZAF98> this is related to a bug in reporting""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1673358651.848909"", ""content"": ""<@U02SCRTM2M7> - I am curious - how permissions in wrong format got created in first place - refer ** - From reporting we dont do create permissions correct ?""}, {""user"": ""sfarooqh"", ""timestamp"": ""1673358799.217099"", ""content"": ""we are doing this create permissions in reporting in order to fetch token from vault-service (by sending these permissions as claims). The reason this may have slipped is that the feature itself does not fail and the report generation is successful, and the logs in reporting service/lambda functions do not show any errors""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1673358869.379309"", ""content"": ""Ohh got it - i am assuming this is nodejs lambda ? Bcoz in java world you can use lib to generate oermissions""}, {""user"": ""sfarooqh"", ""timestamp"": ""1673358909.828369"", ""content"": ""yes this is nodejs lambda""}]" "1681790429.403909 ","[{""user"": ""aganivada"", ""timestamp"": ""1681790429.403909"", ""content"": ""<@U02Q02G09QX> <@U026B1ZN27L> received this notification from openAI. checking if I have access""}, {""user"": ""aganivada"", ""timestamp"": ""1681790572.762129"", ""content"": ""I can submit request to increase quota, <@U0336QZAF98> /<@U02Q02G09QX> any estimate on how much should we request?""}, {""user"": ""aganivada"", ""timestamp"": ""1681790841.726239"", ""content"": ""submitted request to increase quota to 200$ cc: <@U0336QZAF98>""}, {""user"": ""akalyta"", ""timestamp"": ""1681790877.678689"", ""content"": ""<@U02BV2DGUKC> thank you!""}, {""user"": ""aganivada"", ""timestamp"": ""1681893774.536659"", ""content"": ""cc: <@U0336QZAF98> <@U02Q02G09QX>""}, {""user"": ""aganivada"", ""timestamp"": ""1681893897.939629"", ""content"": ""to be on safe side setting hard limit to 150 so we have a 50 buffer in case we crossed the threshold""}, {""user"": ""aganivada"", ""timestamp"": ""1681893905.481039"", ""content"": """"}, {""user"": ""svummidi"", ""timestamp"": ""1682345750.506769"", ""content"": ""Thank you <@U02BV2DGUKC> - I hope you found the credentials in 1Password using operations account.""}, {""user"": ""aganivada"", ""timestamp"": ""1682345832.935869"", ""content"": ""Yes <@U040RCBPBEC> thanks a lot for adding creds in 1 password""}]" "1680516947.597929 ","[{""user"": ""rsrinivasan"", ""timestamp"": ""1680516947.597929"", ""content"": ""<@U02BV2DGUKC> <@U04JT69T00K> - sample for modifying api client moustache- cc: <@U02GC8SE18V>""}, {""user"": ""aganivada"", ""timestamp"": ""1681811192.137459"", ""content"": ""cc: <@U0431DZTPJM>""}]" "1673975093.086339 ","[{""user"": ""aganivada"", ""timestamp"": ""1673975093.086339"", ""content"": ""<@U03KLHDKL1H> did we get a chance to verify this change in int ? we just deployed tenancy with latest commons and getting an exception that put operation is not supported on readonlyhttpheaders\n\n\n\n> java.lang.UnsupportedOperationException\\n\\tat org.springframework.http.ReadOnlyHttpHeaders.put(ReadOnlyHttpHeaders.java:126)\\n\\tSuppressed: The stacktrace has been enhanced by Reactor, refer to additional information below: \\nError has been observed at the following site(s):\\n\\t*__checkpoint \u21e2 Request to GET [DefaultWebClient]\\nOriginal Stack Trace:\\n\\t\\tat org.springframework.http.ReadOnlyHttpHeaders.put(ReadOnlyHttpHeaders.java:126)\\n\\t\\tat com.axm.platform.commons.config.WebClientConfig.lambda$s2sWebClient$0(WebClientConfig.java:102)\\n\\t\\tat\ncc: <@U0431DZTPJM>""}, {""user"": ""aganivada"", ""timestamp"": ""1673975746.108149"", ""content"": ""we seem to be building the object and then updating header field""}, {""user"": ""aganivada"", ""timestamp"": ""1673976406.300299"", ""content"": ""<@U03KLHDKL1H> please review webclientconfig change made as part of to address this issue""}, {""user"": ""askumar"", ""timestamp"": ""1674015522.651479"", ""content"": ""<@U02BV2DGUKC> reviewed the changes for client config ..thanks\nThis particular change was not verified in Int, I was just using it to building app locally, so this runtime error didn't get caught.""}, {""user"": ""aganivada"", ""timestamp"": ""1674015651.321179"", ""content"": ""ok np""}]" "1693198264.002789 ","[{""user"": ""aganivada"", ""timestamp"": ""1693198264.002789"", ""content"": ""<@U03KLHDKL1H> please dont close the tenancy HF yet, I have a couple of fixes , for testing we can deploy tenancy 20.1 to stage""}, {""user"": ""askumar"", ""timestamp"": ""1693198296.927019"", ""content"": ""Sure <@U02BV2DGUKC>""}]" "1692605663.262809 ","[{""user"": ""mnirmal"", ""timestamp"": ""1692605663.262809"", ""content"": ""<@U026PMDB1ND> build is failing in PMS because of tenant-deletion 0.0.1 jar missing - . I have commented the plugin code for now to unblock. Please take a look whenever possible. cc: ""}, {""user"": ""askumar"", ""timestamp"": ""1692606116.635689"", ""content"": ""<@U02SF36PVKL> was this building before?\ncan you try build with latest release version 0.0.2-SNAPSHOT just in case.""}, {""user"": ""mnirmal"", ""timestamp"": ""1692606128.627129"", ""content"": ""tried already, it isn't working""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692638718.076199"", ""content"": ""Hmm okay. That's not supposed to happen. Let me check.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692639984.919189"", ""content"": ""I guess this branch was pulled just before I updated develop. Should have notified widely.\n\nBasically, we need to have a pluginRepositories section in the pom file.\n\nSorry for the lack of info. One way to debug this could be to compare the branch with a passing build and/or to look the readme of the plugin (but the latter would have been hard to locate)""}, {""user"": ""mnirmal"", ""timestamp"": ""1692641464.775569"", ""content"": ""sure, thanks <@U026PMDB1ND>""}, {""user"": ""mnirmal"", ""timestamp"": ""1692737003.686149"", ""content"": ""<@U026PMDB1ND> any reason why this is not in develop yet? I just took another feature branch and had to update this again.\n``` <pluginRepositories>\n <pluginRepository>\n <id>gitlab-maven</id>\n <url></url>\n </pluginRepository>\n </pluginRepositories>```""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692739965.451089"", ""content"": ""Hmm. Is it not?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692739975.569299"", ""content"": ""Let me check""}]" "1680630507.496229 ","[{""user"": ""aganivada"", ""timestamp"": ""1680630507.496229"", ""content"": ""<@U03NZ7Z52S2> Can I deploy tenancy HF on stage to verify the invitation issue?""}, {""user"": ""bganganna"", ""timestamp"": ""1680631623.187439"", ""content"": ""Sure <@U02BV2DGUKC>""}, {""user"": ""aganivada"", ""timestamp"": ""1680631997.123929"", ""content"": ""Thank you Bhavana, will deploy the fix to stage tmrw run some tests and will let you know""}]" "1691507070.318739 ","[{""user"": ""gdahiya"", ""timestamp"": ""1691507070.318739"", ""content"": "" <@U02BV2DGUKC> Tenancy is down. Is there some deployment going on?""}, {""user"": ""askumar"", ""timestamp"": ""1691507126.462529"", ""content"": ""checking <@U042KRZPXHT>""}, {""user"": ""gdahiya"", ""timestamp"": ""1691507184.340949"", ""content"": ""I can see in events that port 8080 was unhealthy and the service got redeployed.""}, {""user"": ""askumar"", ""timestamp"": ""1691507197.668499"", ""content"": ""which env <@U042KRZPXHT>? Int seems to be working""}, {""user"": ""gdahiya"", ""timestamp"": ""1691507209.647739"", ""content"": ""INT only. It is up now.""}, {""user"": ""aganivada"", ""timestamp"": ""1691507231.229779"", ""content"": ""<@U042KRZPXHT> looks like health check failed due to auth0 taking too long to respond""}, {""user"": ""aganivada"", ""timestamp"": ""1691507267.734439"", ""content"": ""checking further""}, {""user"": ""aganivada"", ""timestamp"": ""1691507355.427779"", ""content"": ""service is up now <@U042KRZPXHT>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1691507395.178879"", ""content"": ""<@U02BV2DGUKC> - it is up because auth0 recovered or was some other action required?""}, {""user"": ""aganivada"", ""timestamp"": ""1691507410.166059"", ""content"": ""no <@U026PMDB1ND> auth0 recovered""}, {""user"": ""aganivada"", ""timestamp"": ""1691507418.524809"", ""content"": ""we didnt do anychange from our side""}, {""user"": ""aganivada"", ""timestamp"": ""1691508000.512519"", ""content"": ""we got socket timeouts while reaching auth0 for health checks between 8:29 pm IST to 8:33 pm IST. Though there is no mention in auth0 it does look like some problem . Will raise a support ticket with them to check""}, {""user"": ""aganivada"", ""timestamp"": ""1691508961.255479"", ""content"": ""opened a case with auth0""}]" "1690451262.108649 ","[{""user"": ""aganivada"", ""timestamp"": ""1690451262.108649"", ""content"": "" due to rains in last few days observing frequent network failures, please expect some delay in response. Hopefully it should be resolved by tomorrow.""}, {""user"": ""rvaidya"", ""timestamp"": ""1690460153.375369"", ""content"": ""even had outage\u2026so its okay :slightly_smiling_face:""}, {""user"": ""aganivada"", ""timestamp"": ""1690460674.659349"", ""content"": ""> others cheered as they could take a break from work.\n:astonished:""}]" "1677777497.926499 ","[{""user"": ""sjain"", ""timestamp"": ""1677777497.926499"", ""content"": "",\nI am seeing NPE in INT env while updating pvt (during running integration test)\nReference logs : \n\n\n\nCan somebody help ?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677780725.583249"", ""content"": ""Sanket, let me take a look.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677780915.564209"", ""content"": ""Deploying the current develop version of core""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677784966.653139"", ""content"": ""<@U02TVMF3CR4> - does it work any better now?""}, {""user"": ""sjain"", ""timestamp"": ""1677814643.148239"", ""content"": ""I just checked, It works now <@U026PMDB1ND> Thanks :raised_hands:""}]" "1691509328.712319 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1691509328.712319"", ""content"": ""<@U04JT69T00K> - added scale-gl-runner.sh to release-tools yesterday: \n\nLine 30 in this commit is missing an end-quote. Fixed it in a subsequent commit.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1691509430.217539"", ""content"": ""./scale-gl-runner.sh axmint up force\n./scale-gl-runner.sh axmint down force\n\nWe should see if we can also add a way to poll for the state of the instance and if it is \""stopped\"" after a minute, maybe issue a start.\n\nBasically, avoid the need to do anything via the console, if possible.""}]" "1683618235.853069 ","[{""user"": ""nsrivastava"", ""timestamp"": ""1683618235.853069"", ""content"": "" <@U02D4DUKDQC> the jira integration in gitlab seems to have few broken links, the jira id in MR description does not lead to jira issue directly rather ends up on a `404` page , sample MR . Not sure if this is designed to work this way with some configuration or we need to update the MR template to some thing like this `Jira ID(s): `""}, {""user"": ""aganivada"", ""timestamp"": ""1683619840.812579"", ""content"": ""<@U03RQDE3QUS> can you check the other way around if you mention Jira id in commit message jira should present link to gitlab""}, {""user"": ""nsrivastava"", ""timestamp"": ""1683621047.558799"", ""content"": ""yes <@U02BV2DGUKC>, link in jira seems active and leads to gitlab.""}, {""user"": ""aganivada"", ""timestamp"": ""1683621818.886709"", ""content"": ""Cool I never saw the gitlab link to jira working we can check if there is any setting in gitlab to fix this.""}, {""user"": ""nsrivastava"", ""timestamp"": ""1683622812.603309"", ""content"": ""sure <@U02BV2DGUKC>, thanks.""}, {""user"": ""aganivada"", ""timestamp"": ""1683623407.294809"", ""content"": ""<@U026PMDB1ND> can we update default Web URL to in gitlab jira integration? I tried to do it but I didnt have access to API token for ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683645767.585699"", ""content"": ""<@U02BV2DGUKC> It is already that way. Did someone else update?""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1683648715.224559"", ""content"": ""<@U026PMDB1ND> we need to update Web URL from to currently only th Jira API URL is set to ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683650857.069289"", ""content"": ""oh""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683659667.504379"", ""content"": ""Don't remember how we ended up with this config but it looks like there's a bug in GitLab. It is not working even after the change.\n\nGitLab seems to realize that the issue is a Jira issue (via Regex) but has a problem in constructing the URL.\n\nOther aspects of GitLab -> Jira integration work as expected. For example, commits & branches get recorded as expected in the Jira record.\n\nOpened a case with GitLab. Let's see if there's something else to do to enable the correct URL to be constructed.""}, {""user"": ""aganivada"", ""timestamp"": ""1683689289.174749"", ""content"": ""thank you Rama""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684096340.749659"", ""content"": ""<@U03RQDE3QUS> <@U02BV2DGUKC> <@U02D4DUKDQC> -\n\nIt is working now: \n\nBut it is not retroactive since they seem to save the JIRA link at the time the gitlab message/text is composed""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684096447.483059"", ""content"": ""Nayan, thanks for bringing it up in the first place. We just got used to it not working and went along with it.\n\nThe effect of asking a question should never be underestimated :)""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684096515.832409"", ""content"": ""Now there's more value in having a Jira-id in the MR title.""}, {""user"": ""nsrivastava"", ""timestamp"": ""1684121990.533909"", ""content"": ""Thanks <@U026PMDB1ND>, can see this working. I think wrapping Jira Id in square braces helps Gitlab recognize, leaving it naked does not seem to work. Do we need to update the template?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684122038.018919"", ""content"": ""oh really? Yes, we should ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684124059.807119"", ""content"": ""<@U03RQDE3QUS> - it is working at least in one case. See the description of: ""}, {""user"": ""nsrivastava"", ""timestamp"": ""1684124224.216149"", ""content"": ""ok, not sure <@U026PMDB1ND>, I had checked cases similar to and it was picking only till project. But after updating, started working. same case with MR title.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1684128500.127839"", ""content"": ""ok""}, {""user"": ""nsrivastava"", ""timestamp"": ""1684144567.209469"", ""content"": ""probably those MRs we old or before the change, can see the new MRs () picking up the right navigation. Thanks <@U026PMDB1ND> and please ignore above.""}]" "1691551940.939979 ","[{""user"": ""aganivada"", ""timestamp"": ""1691551940.939979"", ""content"": ""<@U04JT69T00K> can we investigate why this job terminated in runner ? cc: <@U03DHUAJVMK>""}, {""user"": ""pjha"", ""timestamp"": ""1691552067.015709"", ""content"": ""Sire I will check this ""}, {""user"": ""aganivada"", ""timestamp"": ""1691552960.739779"", ""content"": ""<@U03DHUAJVMK> re-running the job completed running all tests ""}, {""user"": ""aganivada"", ""timestamp"": ""1691553022.372469"", ""content"": ""looks like there is some issue with generation of auth0 token\n> Keyword 'Generate Auth0 Token For User' failed after retrying 3 times. The last error was: TypeError: __init__() got an unexpected keyword argument 'chrome_options'""}]" "1679886081.225479 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1679886081.225479"", ""content"": ""<@U03NZ7Z52S2> <@U03DHUAJVMK> - if I add a person to to the team after sending out the pulse, is there a way to include them in the pulse? I don't see them in the respondents list\n\ncc <@U02GC8SE18V>""}, {""user"": ""rvaidya"", ""timestamp"": ""1679887756.928469"", ""content"": ""<@U026PMDB1ND> can you try editing the config(already created) once ?""}, {""user"": ""rvaidya"", ""timestamp"": ""1679888043.133829"", ""content"": ""If that doesnot work, for now you can add the new user with a different persona and send pulse for that persona. <@U026PMDB1ND>""}, {""user"": ""araman"", ""timestamp"": ""1679892329.390949"", ""content"": ""Also we have the new API that was done for EA by <@U02SF36PVKL> to include the latest users added. /api/v1/surveys/{tenantId}/delta-users/send""}, {""user"": ""araman"", ""timestamp"": ""1679892534.192109"", ""content"": ""<@U026PMDB1ND>""}, {""user"": ""mnirmal"", ""timestamp"": ""1679929448.508789"", ""content"": ""API is in SC - ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679940572.409189"", ""content"": ""Thanks <@U02SF36PVKL> and <@U03DHUAJVMK>.\n\nI executed this API. Let's see if the notification goes through.\n\n<@U04EV0M2VA6> - can you please let me know if you received the pulse?""}, {""user"": ""mli"", ""timestamp"": ""1679965872.806389"", ""content"": ""AXM Stage Slack Bot isn\u2019t loading. Last time I checked though (4 hours ago), I hadn\u2019t received the latest Pulse""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679966383.499629"", ""content"": ""<@U04EV0M2VA6> - after you said that it wasn't working (a couple of hours ago), I tried and it worked for me. Can you click on the \""Messages\"" tab and share what you see?""}, {""user"": ""mli"", ""timestamp"": ""1679969514.725969"", ""content"": ""here you go <@U026PMDB1ND>, thanks for looking into this""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679974938.808759"", ""content"": ""Hmm.. <@U04EV0M2VA6> Maybe you can use the Polly survey to record your feedback.\n\n<@U02D4DUKDQC> <@U02SF36PVKL> - can we figure out what happened here? One possibility is the time zone.""}, {""user"": ""mnirmal"", ""timestamp"": ""1679978753.866519"", ""content"": ""\n\nLooks like service user info entry is not present <@U02D4DUKDQC> <@U03BPNY5AGM>""}, {""user"": ""mnirmal"", ""timestamp"": ""1679987602.076519"", ""content"": ""cc: <@U02HQ78V9A5>""}, {""user"": ""pkarthikeyan"", ""timestamp"": ""1679989988.814399"", ""content"": ""<@U02SF36PVKL> service user info entry is lazily loaded. We populate the user when pulse is sent. <@U02D4DUKDQC> Let me know when available we may need to run few queries on stage db to identify the issue.""}]" "1681983020.298699 ","[{""user"": ""ppant"", ""timestamp"": ""1681983020.298699"", ""content"": "" This is a Confluence doc containing the backend OpenAPI generated ApiClient changes and related MRs for accommodating multi instance deployments ""}, {""user"": ""svummidi"", ""timestamp"": ""1682353062.137869"", ""content"": ""<@U0431DZTPJM> How we got the file \u201cApiClient.mustache\u201d - Is it generated file or we created the whole file?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682354360.691269"", ""content"": ""<@U0431DZTPJM> - is there an overall page that describes the approach completely?""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1682356133.068169"", ""content"": ""<@U040RCBPBEC> - this is from openapi tools project -when we openapi to generate cli - it generates based on the same ""}, {""user"": ""svummidi"", ""timestamp"": ""1682357555.979089"", ""content"": ""<@U0336QZAF98> <@U0431DZTPJM> Instead of taking the whole template, is there any way to customize this template? If we really need to take the whole file and modify, it is better to keep the original also in our repository so in future if we need to update it becomes easy to check diffs.""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1682357731.319709"", ""content"": ""Agreed <@U040RCBPBEC> <@U0431DZTPJM> started the poc to see the feasibility of the approach . We can see how we can store in an effective way to handle the use cases you have mentioned""}, {""user"": ""ppant"", ""timestamp"": ""1683004345.258309"", ""content"": ""<@U040RCBPBEC> The template for ApiClient is not generated from Swagger, it uses it automatically. So in order to modify it, we had to take the base template from webclient lib and make changes in it. Ideally, this file will reside in core-libs-data-commons as all the mustache templates are there and we just need to add its dependency in the specification.xml of the projects to use this (along with adding plugin for generating open api client in pom)""}]" "1681280124.499599 ","[{""user"": ""ppant"", ""timestamp"": ""1681280124.499599"", ""content"": "" <@U033PPLNFRU> Starting this thread for discussions regarding controlling invitation emails via feature flags. Right now in Tenancy, the invitation emails are sent from Auth0 and Auth0 doesn\u2019t support multiple templates of the same type.\n\nAs a workaround, we are planning to introduce feature flags, or some param in the API, that can allow us to use our custom templates for sending out invitation email to the users.""}, {""user"": ""aganivada"", ""timestamp"": ""1681281514.529589"", ""content"": ""<@U033PPLNFRU> if you can share content for invitation templates we can cut the route of auth0 invitations and send custom notifications from our side. following are cases where we send invitations today\n\n1. customer sending invite to existing vendor - invitation for account admin\n2. vendor sending invite to existing customer - invitation for product admin\n3. orgadmin inviting a collaborator\n4. SSO migration script triggering invitation to user""}, {""user"": ""anair"", ""timestamp"": ""1681341415.097829"", ""content"": ""noted <@U02BV2DGUKC> <@U0431DZTPJM>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1682010652.004909"", ""content"": ""<@U02BV2DGUKC> - are we waiting on this or is this already in place?""}, {""user"": ""aganivada"", ""timestamp"": ""1682057426.703959"", ""content"": ""We are waiting on this Rama, backend code is in place so once we have template ready we can raise apps ticket to update the api call. Currently we are sending default invitation template from auth0.""}, {""user"": ""anair"", ""timestamp"": ""1683310225.761999"", ""content"": ""<@U026PMDB1ND> <@U02BV2DGUKC> <@U0431DZTPJM> please see (not sure if the wording is appropriate for SSO""}, {""user"": ""anair"", ""timestamp"": ""1683310313.386559"", ""content"": ""(as png's for easier viewing)""}, {""user"": ""anair"", ""timestamp"": ""1683310370.486869"", ""content"": ""figma file btw - ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683310706.399659"", ""content"": ""1. Would we say \""is using Okta for Axiamatic\"" or something more descriptive like \""is using Okta as SSO for Axiamatic\""? \n2. \""Click the button below\"" -> \""Please click the button below\""\n3. \""your strategic partnership\"" seems incomplete to me\n4. \""it's\"" -> its, right?\n Click -> Please click""}, {""user"": ""anair"", ""timestamp"": ""1683326982.225159"", ""content"": ""<@U026PMDB1ND>\n1. Not sure what the right verbage here is - using Okta as SSO for Axiamatic seems redundant since the user will automatically associate Okta with SSO since that is what they use for SSO\n2. Done\n3. \"" {customer} is inviting {vendor} to join Axiamatic to strengthen your partnership\"" sounds simpler than strategic partnership, wdyt?\n4. Done""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1683436380.549149"", ""content"": ""Cool. Sounds good, <@U033PPLNFRU>""}, {""user"": ""aganivada"", ""timestamp"": ""1683520210.172099"", ""content"": ""cool thank you <@U033PPLNFRU> will work with apps team to update the invitation flows. Will create and share an epic for this. cc: <@U0431DZTPJM>""}, {""user"": ""ppant"", ""timestamp"": ""1684303312.378869"", ""content"": ""<@U02BV2DGUKC> <@U033PPLNFRU> <@U026PMDB1ND> For the SSO migration email template, where should \u201cComplete SSO Migration\u201d button take the user? I guess to the Axiamatic login page?""}, {""user"": ""aganivada"", ""timestamp"": ""1684303397.084559"", ""content"": ""yes <@U0431DZTPJM>""}, {""user"": ""aganivada"", ""timestamp"": ""1684303439.345129"", ""content"": ""<@U0431DZTPJM> this is a invitation link url which will take user to login page""}, {""user"": ""aganivada"", ""timestamp"": ""1684303456.844309"", ""content"": ""and if user already has a session with IDP then browser will automatically login""}, {""user"": ""aganivada"", ""timestamp"": ""1684310179.845019"", ""content"": ""<@U02GC8SE18V>, just an FYI platform team is working on a change where we can stop auth0 from sending invitation emails and instead tenancy will send emails through our notification service. This will help us in customising the invitation emails to have more context of our Axiamatic workflow's. The current flow of invitation from auth0 has just one format for all types of invitations.\n\nFrom platform perspective we will add enum's of various templates shared by Aryan above in this link in core-data-commons. We might need apps help to add an additional parameter wherever we are calling tenancy invitation API to pick the correct template. By default if we don't pass any template then we will send notification from auth0 (existing flow) so this is not a breaking change. I will add disc tickets once we have the enum ready.\n\n<@U0431DZTPJM> please add if I missed anything and share sample API request.""}, {""user"": ""ppant"", ""timestamp"": ""1684317691.856939"", ""content"": ""<@U02GC8SE18V> <@U02BV2DGUKC> In the same POST `/api/v1/users/invitation/{tenantId}` API, we have added a new enum field called `invitationType` in the API request body (defaults to INVITE which triggers Auth0 to send the invitation email) in which you can pass values that indicate which templates to use. Currently only `SSO_REINVITE` is there for SSO migration, will be adding more accordingly. Here is a sample request body\n```{\n \""emails\"": [\n \""string\""\n ],\n \""role\"": \""orgAdmin\"",\n \""inviter\"": \""string\"",\n \""invitationType\"": \""INVITE\"",\n \""permissions\"": [\n {\n \""resourceTenantId\"": 1234,\n \""resourceType\"": \""productInstance\"",\n \""resourceId\"": \""19\"",\n \""allowedOperation\"": \""read-write\"",\n \""relationType\"": \""customer\""\n }\n ]```""}, {""user"": ""rvaidya"", ""timestamp"": ""1684338101.366779"", ""content"": ""<@U02BV2DGUKC> <@U0431DZTPJM> Ack for the note and taking care with default value.\nFeel free to create the jiras and assign it to me and we can pick this within the team.""}, {""user"": ""aganivada"", ""timestamp"": ""1684338123.835789"", ""content"": ""sure thank you <@U02GC8SE18V>""}, {""user"": ""aganivada"", ""timestamp"": ""1684343156.695669"", ""content"": ""<@U02GC8SE18V> added for this task. I've added one task please split based on the flows we support from dashboard in required.\n\n1. customer sending invite to existing vendor - invitation for account admin\n2. vendor sending invite to existing customer - invitation for product admin\n3. orgadmin inviting a collaborator\n4. _SSO migration script triggering invitation to user - not required for apps flow this is a backend only flow when we perform SSO migration_\n""}, {""user"": ""rvaidya"", ""timestamp"": ""1684379376.104089"", ""content"": ""<@U02BV2DGUKC> the below falls with default invite \u2026 right?\n> orgadmin inviting a collaborator ""}, {""user"": ""aganivada"", ""timestamp"": ""1684380494.045949"", ""content"": ""yes <@U02GC8SE18V> this is the collaborators invitation flow""}, {""user"": ""aganivada"", ""timestamp"": ""1684380847.541339"", ""content"": ""<@U033PPLNFRU> we just completed backend work for the email templates can this be taken up for apps integration in 0.9.12? cc: <@U02GC8SE18V>""}]" "1684818084.842979 ","[{""user"": ""pjha"", ""timestamp"": ""1684818084.842979"", ""content"": ""<@U02BV2DGUKC> <@U026PMDB1ND> please review multi-instance-deployment phase-1 changes from infra side; enabled for experiment-service and tenancy-service ""}, {""user"": ""aganivada"", ""timestamp"": ""1684818159.471729"", ""content"": ""<@U04JT69T00K> please post reviews in <#C02BXE5HS00|merge-requests>""}]" "1684926828.611679 ","[{""user"": ""hchintamreddy"", ""timestamp"": ""1684926828.611679"", ""content"": ""<@U02BV2DGUKC> We have an issue with stuck deployment in INT for pulse manager can you help with it?""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1684927833.255649"", ""content"": ""*Task stopped at: 24/05/2023, 11:29:56 UTC*\n`CannotPullContainerError: pull image manifest has been retried 1 time(s): failed to resolve ref : : not found`""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1684927841.447639"", ""content"": ""cc <@U02TVMF3CR4>""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1684928898.121159"", ""content"": ""<@U02TVMF3CR4> have you pulled latest develop branch for CDK deployment before deploying? I have killed the current deployment and restarted it checking if it succeeds""}, {""user"": ""sjain"", ""timestamp"": ""1684929215.972089"", ""content"": ""I did took pull couple of days back..let me retry""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1684929225.005329"", ""content"": ""No need right now""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1684929234.279819"", ""content"": ""<@U02TVMF3CR4> I have initiated one already""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1684929276.834699"", ""content"": ""It looks like the image name has pulse-manger twice which could have been the problem from above error""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1684929352.582239"", ""content"": ""deployed successfully""}, {""user"": ""aganivada"", ""timestamp"": ""1684932140.410279"", ""content"": ""Sorry <@U02D4DUKDQC> just saw this message""}, {""user"": ""hchintamreddy"", ""timestamp"": ""1684936482.318159"", ""content"": ""no worries <@U02BV2DGUKC> was able to fix it""}]" "1673872682.157089 ","[{""user"": ""ppant"", ""timestamp"": ""1673872682.157089"", ""content"": ""<@U02BV2DGUKC> Regarding pagination for get customers API `/api/v1/link/vendor/{tenantId}`, there are two calls that happen to core data -\n\n1. When we have the invites present in `props` column in tenant table for that tenant, we call core data to get customers for each invite.\n2. Then a call to core-data is made to get all the customers for this tenant. For each customer, we get the tenant and populate the details.\nNow I have implemented pagination for `/api/v1/vendor/{tenantId}/customers` API using a list and its indexes to maintain page number and size as it only makes the first call but we won\u2019t be able to sync the page number and size with the second call as it will be hard to maintain the number of results got in the first call with the second when going back and forth on the pages.\n\nOne easy way I could think is that in the backend we get all the data but in the API we present only the slice that in accordance with the pagination provided. Does this sound good?""}, {""user"": ""aganivada"", ""timestamp"": ""1673888229.745769"", ""content"": ""<@U0431DZTPJM> if we split the ui to two components invitations and active customers would that make it easy? I am not against consolidation of responses but the complication is we'd endup retrofitting 2 different types of responses.""}, {""user"": ""aganivada"", ""timestamp"": ""1673889137.739609"", ""content"": ""Prabhu I see the issue you are talking about, the V1 model is looking for customers from props is deprecated now, we migrated vendors (axiamatic only as all new customers are in V2 model) to new model so we don't have to worry about 1. Will check in prod once to confirm. Let's just look at pagination of 2. However the bigger problem is consolidation of the responses in ui, let's discuss with <@U02GC8SE18V> on this tmrw. Cc: <@U026PMDB1ND>""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1673893231.831019"", ""content"": ""<@U0431DZTPJM> please let me know once you make the changes? I need to update the admin app as well""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1673909134.746219"", ""content"": ""<@U02BV2DGUKC> - I only have a partial understanding of the problem but if we can add an API to core-data to narrow what we are looking for instead of returning everything, we should consider it.\n\nIs there a way we can get all the tenants of a page and craft a rsql query predicate (in clause) and call to get invitation contexts for specific customers only?""}, {""user"": ""aganivada"", ""timestamp"": ""1673928899.540429"", ""content"": ""hmmm...... makes sense <@U026PMDB1ND> will discuss with Ritu and Prabhu on this and check if we can optimize the call by querying only invitations, for the context today when we show customers we have 2 sections:\n\n1. customers who were invited by vendor but customer has not accepted invitation (core): this information is only in core data invitation context table as there is no customer tenant id or such it is just an invitation sent to customer. Apps is making this call directly to core-data to fetch open invitations via rsql.\n2. customers who have accepted the invitation (tenancy): here the source is customer-vendor mapping table in core which was created when customer and vendor are linked during onboarding. in this case tenancy fetches the info and enriches it with organization details and forwards it to apps. We need the organization info to construct magic link so users can switch to customers org. \n since the sources were 2 different tables pagination became tricky. Based on the suggestion I guess what we could do is filter all the invitations on a given tenant say from tenancy itself and updated the records with organization info wherever applicable.""}, {""user"": ""ppant"", ""timestamp"": ""1674557993.001149"", ""content"": ""<@U0281D3GWHL> FYI have deployed this to int as a new API and here\u2019s a sample curl\n```curl -X 'GET' \\\n '' \\\n -H 'accept: application/json' \\\n -H 'Authorization: Bearer <token>'```""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1674558862.695709"", ""content"": ""Thanks, I will take a look <@U0431DZTPJM>""}]" "1692405461.473719 ","[{""user"": ""svummidi"", ""timestamp"": ""1692405461.473719"", ""content"": ""<@U026PMDB1ND> - For production release with hot-fixes, what is the recommended practice? Do we need to merge the branch before release or wait until deployment and then merge?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692405827.838289"", ""content"": ""Since we'd be testing the HF build in int and stage, we could deploy that and then merge. Otherwise there's a small chance that what we tested might be different from what we deploy.\n\nOTOH, if two HFs are running in parallel deploying from one HF could lose the changes in the other.\n\nThough it takes one more round of validation (if there are any diffs), deploying from main is the safest thing to do.\n\n<@U02BV2DGUKC> - are we following this approach?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692405970.477419"", ""content"": ""<@U040RCBPBEC> - if\n\ngit diff main..hofix/x.x.x.x shows only the changes on the HF side, merge to main (hotfix finish) so that we can deploy the main. Deploy from HF only if we are not sure.""}, {""user"": ""svummidi"", ""timestamp"": ""1692408736.134729"", ""content"": ""<@U026PMDB1ND> for now we all tested from hf branch only and deployment starts in few hours, so I will defer the merge to main to Monday. Next time, we will plan better to follow the recommended practice.""}, {""user"": ""aganivada"", ""timestamp"": ""1692418018.399309"", ""content"": ""> are we following this approach?\nyes <@U026PMDB1ND> as of today during testing int and stage we are deploying from hf branch once we are done with tests we are merging to main and develop (with occasional re-tests after closing hf if required) and then deploying to prod from main. This ensures deployment happens after closing the hf.""}, {""user"": ""gdahiya"", ""timestamp"": ""1692515816.252679"", ""content"": ""> OTOH, if two HFs are running in parallel deploying from one HF could lose the changes in the other.\n<@U026PMDB1ND> In apps we always create one HF branch and pull out a branch from the HF branch for every task/fix we work on and then merge this branch back to HF. This way the HF branch is always up to date and we don\u2019t have multiple HFs running in parallel. We merge the HF to main a day or two before prod deployment.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1692564212.983549"", ""content"": ""This is good, <@U042KRZPXHT> Since gitflow doesn\u2019t support this, do you manually update the pom file versions in the sub-branches and adjust the same when merging back? Or, is there a better way to do this?""}, {""user"": ""gdahiya"", ""timestamp"": ""1692622892.181729"", ""content"": ""<@U026PMDB1ND> If we want to deploy the specific change and test it on INT or STAGE we update the pom versions manually, otherwise we just keep the pom version same as in the HF branch and always deploy the HF branch after merging the task branch (in this case it makes it easier for others to review the specific change as we can raise the MR against the HF branch).""}]" "1679070698.108989 ","[{""user"": ""ppant"", ""timestamp"": ""1679070698.108989"", ""content"": "" Here is the confluence doc for Lambda integration with VPC ""}, {""user"": ""aganivada"", ""timestamp"": ""1679286533.774409"", ""content"": ""thank you <@U0431DZTPJM>""}, {""user"": ""aganivada"", ""timestamp"": ""1679291045.507989"", ""content"": "" one of the requirements of VPN is to have our lambda code updated to make API calls through private domain instead of using API gateway this is required since we will be restricting public access to all backend services. Please review document prepared by <@U0431DZTPJM> on changes required for existing lambda functions.\n\nPlatform team will update common libraries and shared lambdas we will need your help in updating lambda's that you own . We can discuss more on this during tmrw's dependency call.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679291269.508679"", ""content"": ""<@U02BV2DGUKC> - to make things move faster, if the changes are straightforward, one option is for the platform team to make the changes and get them reviewed. Did we consider that option?""}, {""user"": ""aganivada"", ""timestamp"": ""1679291782.571209"", ""content"": ""sure <@U026PMDB1ND>, we did consider it but after reviewing the lambdas we noticed some lambda's are not using common libraries so there might be some additional work required and considering the domain knowledge we thought it might be faster if respective team looks into it, wherever lambdas are using common libraries changes should be straight forward. Either ways we will need individual teams help in reviewing and testing changes once we update the lambda.""}, {""user"": ""aganivada"", ""timestamp"": ""1679291832.740119"", ""content"": ""overall we need help with 3 lambdas other 4 Platform team will look into updating ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679291922.644429"", ""content"": ""If we can get all the lambdas to start using the common lib it could be a desirable side-effect""}, {""user"": ""askumar"", ""timestamp"": ""1679292079.801519"", ""content"": ""<@U02BV2DGUKC> there is one Apps Workflow Lambda , can we include that as well please.""}, {""user"": ""aganivada"", ""timestamp"": ""1679292172.410949"", ""content"": ""sure thank you <@U03KLHDKL1H>""}, {""user"": ""aganivada"", ""timestamp"": ""1679292214.096299"", ""content"": ""<@U03KLHDKL1H> will assign this to you since you have good amount of knowledge on this lambda""}, {""user"": ""askumar"", ""timestamp"": ""1679292246.962849"", ""content"": ""yes thankyou <@U02BV2DGUKC> , I will make changes for it.""}]" "1688460874.469649 ","[{""user"": ""askumar"", ""timestamp"": ""1688460874.469649"", ""content"": "" <@U026PMDB1ND> <@U02BV2DGUKC>\n\nWe have updated the maximum ACU for INT Aurora cluster to 5 ACUs, because of the high CPU utilisation.\n\n\nThanks""}, {""user"": ""aganivada"", ""timestamp"": ""1688461394.879449"", ""content"": ""<@U03KLHDKL1H> weren't we planning to make it 5ACU ? may be once we have proxy configured we can monitor better""}, {""user"": ""askumar"", ""timestamp"": ""1688461444.144789"", ""content"": ""Yeah my bad, updating it. Thanks <@U02BV2DGUKC>""}]" "1677841959.789749 ","[{""user"": ""pjha"", ""timestamp"": ""1677841959.789749"", ""content"": ""Hi , please let me know if you have any lambda function which communicate with the any of our service, as we are planning to make few modifications around the service call endpoints.\nrefer : ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677868317.676059"", ""content"": ""<@U04JT69T00K> - it is a bit complicated but tenant deletion the most comprehensive lambda in terms of invoking various services.\n\n`platform-lambda-coord-invoker/src/platform_lambda_coord_invoker/lambda_function.py`\n\n```126 url = f\""{app_config.get_api_domain_url()}/{event['service']}/{get_uri(event_type, event)}\""```\n\napp_config.get_api_domain_url() comes via Lambda env\n\n```platform-libs-epoch/package/platform_libs_epoch/configuration/app_config.py\n\n 69 def get_api_domain_url(self):\n 70 return settings.get(\""API_GW_URL\"")```\nIf we replace {app_config.get_api_domain_url()} with something service-specific we'd be able to cover almost all the services in one shot.\n\nIn the above fragment, event['service'] is configured via coordination_building_blocks/step_function_builder_stack.py in cdk-artifacts and in the state machine definition (apps/coordination-building-blocks/state-machines/tenancy-tenant-deletion.json).\n\nI see platform-lambda-triggerpulse and platform-lambda-useraction also making calls to the services via the API GW. They too use platform-libs-epoch's app-config which in turn depends on API_GW_URL.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677868752.728999"", ""content"": """"}, {""user"": ""pjha"", ""timestamp"": ""1678171050.131839"", ""content"": ""<@U026PMDB1ND> sure I will go through it""}]" "1678254053.890079 ","[{""user"": ""ppant"", ""timestamp"": ""1678254053.890079"", ""content"": "" If we are looking for a good tool that provides BI with visualisation and an interface for executing SQL on selected DBs and tables, we can also consider Apache Superset and Metabase\n\n""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678254117.018479"", ""content"": ""We should prefer things that we don't have to run ourselves.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1678254124.910909"", ""content"": ""In most cases...""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678262270.783189"", ""content"": ""Imply druid has a very good dashboard builder plus Microsoft powerbi""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1678297636.224819"", ""content"": ""<@U02BV2DGUKC> - If we are planning to shift to powerbi or druid - right now we do a metric based approach - where micro service keeps track of number , prometheus scrapes it and send it - Instead we can have an event driven approach - where micro service just post an event with context- tenantId , userid , to bus - kinesis or sqs - from there we can directly send to powerbi or druid or even google GA - we can build dashboard on top of these events - just food for thought\n```{\n\""event:\""pulse created\"",\n\""id\"":\""\"",\n\""time\"":\""\""\n}```""}, {""user"": ""aganivada"", ""timestamp"": ""1678335400.588309"", ""content"": ""yup agreed thank you <@U0336QZAF98>, Rama also had a similar point we should split the data to static content from db that drona HQ/retool etc and stream event data to analytical system like snowflake so we can attach time series and be able to write simple sql queries for visualization.\n\nIssue with current model is with scraping prometheus attaches the time it scrapped and there is no way to change it (known limitation) also mapping id's to name's is way too much overhead on services.""}]" "1687871035.398049 ","[{""user"": ""aganivada"", ""timestamp"": ""1687871035.398049"", ""content"": "" <@U026PMDB1ND> <@U02GC8SE18V> <@U02HCMTQU3W> <@U028EDANJM9> For CISO dashboard we have a scenario where we need to fetch product admins based on solution id/id's. This info is required to list owners on solutions landing page for project/initiative (screenshot attached). Since we don't store users, roles and permissions in our database fetching this info directly may be tricky so in order to address this I could think of 2 approaches:\n\n1. Approach-1: we use auth0 query API to search for product owners based on user metadata. So lets say we wanted to list all product owners for product instance id 11071 of tenant id 19190 then we can trigger a call to auth0 list users api with q as `user_metadata.axm_permissions: (\""19190::c::19190:pin:11071::rw\"")` this will fetch all the users where users permissions explicitly gives them read write access to product instance `11071` and does not list other enterprise admins who get rw access to the product due to super user privilege. Even if we have multiple products we can request a comma separated list of permissions `user_metadata.axm_permissions: (\""19190::c::19190:pin:11071::rw\"", \""49286::c::49286:pin:30324::rw\"")` with one API call. We can wrap these calls with spring redis cache so we don't have to reach auth0 every time and update cache whenever there is a permission change at tenant/product instance level\n2. Approach-2: make core changes to maintain a DB table with web-app access users, their roles and link product instances they own. Tenancy will maintain the data and return results directly from core instead of querying auth0. Tenancy need to also make changes to keep both auth0 and our DB consistent. This came up in another discussion also on should we maintain a replica of roles and permissions in our database as well. \nSince API specs need not change and client should not be worried about where the data is coming from (as long as SLA's are reasonable) for the simplicity and speed of execution I think we can start with approach-1 for CISO dashboard. and Approach-2 can be tracked as an independent item so we get sufficient time to research on where or how to store permissions. Please let me know if it makes sense.""}, {""user"": ""rvaidya"", ""timestamp"": ""1687874271.897609"", ""content"": ""<@U02BV2DGUKC> i agree with you to get an API in tenancy with approach 1 and later change it to approach 2 (which can be a longer lead time).""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1687874752.465799"", ""content"": ""I agree with Ritu. We should start on (2) as well if we have cycles because (1) doesn\u2019t sound like a long term option.""}, {""user"": ""aganivada"", ""timestamp"": ""1687875647.263609"", ""content"": ""got it, thank you <@U026PMDB1ND> & <@U02GC8SE18V> will add an epic for (2) but it wont be part of CISO dashboard. Once we have (2) in a decent shape and migration jobs to preload data for existing customers users we can use FF to switch to (2) but the API spec should not change.""}, {""user"": ""aganivada"", ""timestamp"": ""1689316109.708619"", ""content"": ""cc: <@U03KLHDKL1H>""}]" "1683644993.173979 ","[{""user"": ""aganivada"", ""timestamp"": ""1683644993.173979"", ""content"": ""<@U0431DZTPJM> <@U04JT69T00K> jfyi deleting tenancy multi-instances deployed in int for now, we can revisit while testing multi-instance changes""}, {""user"": ""aganivada"", ""timestamp"": ""1683645266.297789"", ""content"": ""<@U04JT69T00K> do you want to keep *exp-stack-int-plat-1 for validation? I can reduce the desired instances to 0 for now*""}, {""user"": ""pjha"", ""timestamp"": ""1683645329.457549"", ""content"": ""Yes, please reduce it to 0. ""}]" "1680059545.503019 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1680059545.503019"", ""content"": "" - did we make any changes to prod ES () recently?\n\nLooks like the node count fell from 7 to 4?\n\nfyi <@U028EDANJM9>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680059643.228019"", ""content"": ""The other cluster (*oses-plat-evtmgmt-production)* is green""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680059968.266819"", ""content"": ""<@U028EDANJM9> - didn't we discuss that we can probably turn the *oses-plat-evtmgmt-production* cluster off since we are not searching anything from here? If it is just for backup, we could just write to s3 in batches and then clean up periodically. Unless I am misremembering things...\n\nWe are spending quite a lot now.""}, {""user"": ""gshenoy"", ""timestamp"": ""1680060098.020409"", ""content"": ""Yes <@U026PMDB1ND>, that is correct. Will make the necessary changes to phase this out. \nCc <@U02HCMTQU3W>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680060123.501819"", ""content"": ""9338 is roughly 14 (nodes) * 28 (days in march) * 24h""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680060172.926079"", ""content"": ""<@U028EDANJM9> - we can probably reduce its capacity for now since we won't query""}, {""user"": ""gshenoy"", ""timestamp"": ""1680060235.041489"", ""content"": ""Sure <@U026PMDB1ND> . Will do that right away. By 0.9.10, we can even remove this.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680060287.795809"", ""content"": ""Cool. That'd be roughly $500/month of savings""}, {""user"": ""aganivada"", ""timestamp"": ""1680061419.815699"", ""content"": ""> did we make any changes to prod ES () recently?\nno <@U026PMDB1ND> not that I am aware of. we did deploy a HF builds of MB, pulse-manager and pusel-survey app around the time total nodes came down but not sure if they'd have any impact on total nodes of this cluster""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680064143.011819"", ""content"": ""Hmm.. it is very weird.""}, {""user"": ""aganivada"", ""timestamp"": ""1680070596.053439"", ""content"": ""cluster config still says 7 nodes (4 data + 3 master) but there seem to be only 2 data nodes + 3 mater nodes running. Cluster health also seem to be suggesting there are 4 nodes instead of 5""}, {""user"": ""gshenoy"", ""timestamp"": ""1680089131.070039"", ""content"": ""<@U026PMDB1ND> *oses-plat-evtmgmt-production* cluster size is downsized in prod.\nThank you <@U02BV2DGUKC> for helping with the config and deployment.\n\nCreated to remove this dependency completely.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680103204.782969"", ""content"": ""Thanks <@U028EDANJM9> ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680103219.114229"", ""content"": ""<@U02BV2DGUKC> did you open a case?""}, {""user"": ""aganivada"", ""timestamp"": ""1680103404.179019"", ""content"": ""no <@U026PMDB1ND> not yet, will open one""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680103444.188059"", ""content"": ""please cc me""}, {""user"": ""aganivada"", ""timestamp"": ""1680104525.651699"", ""content"": ""done <@U026PMDB1ND> ""}, {""user"": ""gshenoy"", ""timestamp"": ""1680247080.122979"", ""content"": ""<@U026PMDB1ND> do you think it is useful to archive these events in s3 or do we just cut the chord and not persist them ?\n\nKinesis allows to set the destination as s3 directly. We can do that instead of the current model of elastic search with s3 backup.\n\nIf we are never going to query these events, we can just let the services consume them and not have a catch all filter in sns.\nCc <@U02HCMTQU3W>""}, {""user"": ""rsrinivasan"", ""timestamp"": ""1680247189.647529"", ""content"": ""my vote for s3 archival - as it will be helpful in debugging issues - when we have to look at what events received""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680282618.059629"", ""content"": ""Can we set a TTL and let things expire if we write to S3?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680282634.588279"", ""content"": ""If not, we need to find a way to delete things in S3""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680282707.571109"", ""content"": "" - looks like we should be able to do it""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1680282835.400379"", ""content"": ""Getting it done via CDK is probably a bit of work, but we can start by directing the events to S3 and work on setting up the expiration policies. The TTL could be 30 days in prod (to match logz), and 7 days in INT and Stage. We can monitor the cost and figure out.""}, {""user"": ""aganivada"", ""timestamp"": ""1680499247.718389"", ""content"": ""I think if we create s3 through CDK we might be able to set object-expiration. <@U04JT69T00K> recently did this for the s3 bucket which stores cloudfront logs""}, {""user"": ""pmangalapuri"", ""timestamp"": ""1680502702.362409"", ""content"": ""sure <@U026PMDB1ND>, we already have a CDK code for kinesis to s3 for data team requirements. We can leverage it. There might be a few modifications though. will check and update. <@U028EDANJM9>""}]" "1677475296.106199 ","[{""user"": ""snangia"", ""timestamp"": ""1677475296.106199"", ""content"": "" do we have a cdk support to update permission policies for Gitlab-Agent-Role or it needs to be done manually from console?""}, {""user"": ""askumar"", ""timestamp"": ""1677475449.857049"", ""content"": ""~<@U04JT69T00K> do you have an idea about this ?~""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677475466.012439"", ""content"": ""We don\u2019t have a way""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677475488.317239"", ""content"": ""<@U03RSS0S76Y> - what is needed in this case?""}, {""user"": ""aganivada"", ""timestamp"": ""1677475532.156559"", ""content"": ""<@U026PMDB1ND> we need access to a s3 bucket similar to the notification-data bucket""}, {""user"": ""pjha"", ""timestamp"": ""1677475532.282459"", ""content"": ""Not aware of this""}, {""user"": ""aganivada"", ""timestamp"": ""1677475539.431289"", ""content"": """"}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677475573.235529"", ""content"": ""<@U02BV2DGUKC> can you take care of this?""}, {""user"": ""aganivada"", ""timestamp"": ""1677475589.727149"", ""content"": ""sure <@U026PMDB1ND>, I will take care of it""}, {""user"": ""snangia"", ""timestamp"": ""1677475606.855799"", ""content"": ""Thanks <@U02BV2DGUKC> <@U026PMDB1ND>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677475627.071059"", ""content"": ""Converting this to cdk could be something we should try""}, {""user"": ""aganivada"", ""timestamp"": ""1677475668.153719"", ""content"": ""yeah will add a task for this""}, {""user"": ""aganivada"", ""timestamp"": ""1677475716.702089"", ""content"": ""<@U03RSS0S76Y> please share the bucket details once deployed I can add permissions for the gitlab-user role""}, {""user"": ""snangia"", ""timestamp"": ""1677477890.982989"", ""content"": ""<@U02BV2DGUKC> please add s3 read write permission for in INT env.""}, {""user"": ""aganivada"", ""timestamp"": ""1677491427.846909"", ""content"": ""<@U03RSS0S76Y> updated int policy to include Get,Put & Delete. Please check and let me know if it works for your case then will update policy on stage and prod""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677514917.640829"", ""content"": ""<@U03RSS0S76Y> - do we need this in non-INT envs as well?""}, {""user"": ""snangia"", ""timestamp"": ""1677515768.818279"", ""content"": ""yes <@U026PMDB1ND>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677516236.721739"", ""content"": ""Do we already have the resources against which these policies need to be written, <@U03RSS0S76Y>?""}, {""user"": ""snangia"", ""timestamp"": ""1677561656.326889"", ""content"": ""works <@U02BV2DGUKC> :+1:""}, {""user"": ""snangia"", ""timestamp"": ""1677561686.521969"", ""content"": ""<@U02BV2DGUKC> please hold on for other env, it's still in POC mode.""}, {""user"": ""aganivada"", ""timestamp"": ""1677563616.856229"", ""content"": ""sure""}, {""user"": ""aganivada"", ""timestamp"": ""1677563726.638549"", ""content"": ""<@U03RSS0S76Y> teams-bot s3 bucket was created manually earlier to store the bot zip files, as part of your change are we planning to add cdk change for creating a s3 bucket also? (this is in continuation to Rama's question on this thread) cc: <@U02D4DUKDQC>""}, {""user"": ""snangia"", ""timestamp"": ""1677580546.491959"", ""content"": ""we already have the s3 buckets in all env, we just need to push the latest version of app to existing buckets which will be handled using aws cli itself.""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1677599658.266519"", ""content"": ""okay""}, {""user"": ""snangia"", ""timestamp"": ""1678206847.099599"", ""content"": ""<@U02BV2DGUKC> can you please make the changes for S3 buckets write access in stage and prod aws console as well\nAlso need to edit the trust policy to add `project_path:axiamatic-main/collaboration-services-teams:ref_type:branch:ref:*`""}, {""user"": ""aganivada"", ""timestamp"": ""1678245760.704619"", ""content"": ""<@U04JT69T00K> can we look into this?""}, {""user"": ""pjha"", ""timestamp"": ""1678254377.023679"", ""content"": ""<@U02BV2DGUKC> I will make the change""}, {""user"": ""pjha"", ""timestamp"": ""1678256438.769279"", ""content"": ""There is no such bucket in staging""}, {""user"": ""pjha"", ""timestamp"": ""1678256527.430699"", ""content"": ""<@U02BV2DGUKC> <@U03RSS0S76Y> I have added project_path:axiamatic-main/collaboration-services-teams:ref_type:branch:ref:* in the trust policy for staging""}, {""user"": ""pjha"", ""timestamp"": ""1678256571.748999"", ""content"": ""<@U02BV2DGUKC> I don't have prd access.""}, {""user"": ""aganivada"", ""timestamp"": ""1678256717.493729"", ""content"": ""<@U04JT69T00K> you should have prod access we enabled it last week right?""}, {""user"": ""aganivada"", ""timestamp"": ""1678256737.415269"", ""content"": ""what error do you see when logging into axm-prod?""}, {""user"": ""pjha"", ""timestamp"": ""1678256769.577029"", ""content"": ""<@U02BV2DGUKC> o sorry, I should have it""}, {""user"": ""pjha"", ""timestamp"": ""1678256916.165549"", ""content"": ""I am able to login, my bad""}, {""user"": ""pjha"", ""timestamp"": ""1678257078.313059"", ""content"": ""bucket is not there in prd""}, {""user"": ""aganivada"", ""timestamp"": ""1678258033.792249"", ""content"": ""<@U04JT69T00K> it should be under static content bucket""}, {""user"": ""pjha"", ""timestamp"": ""1678259922.844819"", ""content"": ""s3 access permissions added to both stage and prd""}, {""user"": ""snangia"", ""timestamp"": ""1678291758.704979"", ""content"": ""Thanks <@U04JT69T00K>""}]" "1672856401.708739 ","[{""user"": ""rtaraniganty"", ""timestamp"": ""1672856401.708739"", ""content"": ""<@U02BV2DGUKC> found this re: how to connect to VPN from GitLab programmatically - \n\nTalks about OpenVPN""}, {""user"": ""ppant"", ""timestamp"": ""1672888273.197229"", ""content"": ""<@U026PMDB1ND> Just for my knowledge, what kind of GitLab jobs would require access to resources in VPN? I guess it is cross application tests or something like that?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1672888343.704179"", ""content"": ""Yeah, system test jobs that talk to the service end points using s2s token""}, {""user"": ""aganivada"", ""timestamp"": ""1672892049.410579"", ""content"": ""awesome!! really cool thank you Rama""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1672903819.568529"", ""content"": ""<@U02BV2DGUKC> - I spent a bunch of time today talking to Perimeter 81 support people. I didn't get any useful answers, but they setup a call at 11:30 am PST tomorrow where we may get more info.\n\nI don't know if the article cited above could work with Perimeter 81.\n\nMeanwhile, there could be other ways of securing access to our APIs. One simple way is to add the temporary creds we get from aws sts-assume role as a header for all the requests that our tests invoke and the authorizer could turn around and call to validate the header.\n\nBasically programmatic equivalent of:\n\n% export AWS_ACCESS_KEY_ID=...\n% export AWS_SECRET_ACCESS_KEY=...\n% export AWS_SESSION_TOKEN=...\n\nwhich are in the response value returned by:\n\n```STS_RESPONSE=($(aws sts assume-role-with-web-identity \\\n --role-arn ${env_role_arn} \\\n --role-session-name \""SysTestRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}\"" \\\n --web-identity-token $CI_JOB_JWT_V2 \\\n --duration-seconds 2400 \\\n --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \\\n --output text))\n\n(we use this today)```\n% *aws sts get-caller-identity*\n*{*\n \""UserId\"": \""AROAWKADOJDO56NC2GY3Y:SysTestRunner-30887682-738783884\"",\n \""Account\"": \""433798924509\"",\n \""Arn\"": \""arn:aws:sts::433798924509:assumed-role/Gitlab-Agent-Role/SysTestRunner-30887682-738783884\""\n}\n\nWe can then check the Arn to see that it starts-with _arn:aws:sts::433798924509:assumed-role/Gitlab-Agent-Role/SysTestRunner-_ and allow access.\n\nNo one would be able to get that token except a job running on GitLab. Also, the token's life time would be very short (40 mins today).""}, {""user"": ""aganivada"", ""timestamp"": ""1672906389.743719"", ""content"": ""Ok, will give this a try Rama.""}, {""user"": ""aganivada"", ""timestamp"": ""1672914306.237079"", ""content"": ""<@U026PMDB1ND> so should authorizer validate by creating a client with session tokens passed in header -> invoke *get-caller-identity -> look at the arn and then allow? is it safe to pass the creds in headers?* ""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1672940297.492249"", ""content"": ""They do get passed over https to AWS api endpoints when we invoke any AWS commands.\n\nLet's see what Perimeter81 folks say today""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1672948883.263269"", ""content"": ""Debrief of the call with Perimeter 81:\n\n1. They offer IPSEC tunnels or OpenVPN tunnels. I need to read up on the differences, but the article at the top of this thread relies on OpenVPN, so maybe it can work.\n2. We can use one instance of a Gateway, create 3 tunnels, 1 per AWS env. We would also create 3 groups of users and bind each group to a tunnel. Since users can belong to multiple groups, group membership would determine which tunnel one is allowed to join. \n3. Pricing is per user per month.\n4. Monthly and annual plans exist. With annual plan you can't reduce the licenses you already bought but monthly allows you to add and delete (is not a likely concern for us)\n5. They don't have a free trial period, but have a 30 day money back guarantee. So, once we sign up we'd have 30 days to complete all POCs/validations. \nfyi <@U02BV2DGUKC> <@U0281D3GWHL>""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1672948986.913869"", ""content"": ""<@U02BV2DGUKC> - can we build an authorizer that just logs the IP address of the caller and attach it to the GW in INT? Doesn't need to make any disallow decisions for now. Just monitor mode would do.""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1672966673.043649"", ""content"": ""Will we go with AWS or O365 for Auth <@U026PMDB1ND>?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1672966754.268039"", ""content"": ""O365 from what I can tell. Raghu would know""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1672966776.180759"", ""content"": ""Because we don't have one complete source of truth in AWS.""}, {""user"": ""psomasamudram563"", ""timestamp"": ""1672966776.699019"", ""content"": ""ok thanks""}, {""user"": ""aganivada"", ""timestamp"": ""1672978108.312869"", ""content"": ""> can we build an authorizer that just logs the IP address of the caller and attach it to the GW in INT? Doesn't need to make any disallow decisions for now. Just monitor mode would do.\n<@U026PMDB1ND> we are already getting this info on logz via clientip header can we use this for monitoring? cc: <@U0336QZAF98> <@U03KLHDKL1H>""}]" "1679400285.396849 ","[{""user"": ""ppant"", ""timestamp"": ""1679400285.396849"", ""content"": ""<@U026PMDB1ND> Wanted to check on one thing regarding `AxmFilter`. Sometimes for tenant specific logging if we have mentioned level in the filter as DEBUG, we are also seeing some TRACE logs. I was going through the code of this again and noticed in some `filter()` methods we have used instead of . Is this an expected condition because I think this is letting lower level logs to pass through the filter?""}, {""user"": ""rtaraniganty"", ""timestamp"": ""1679422041.706529"", ""content"": ""<@U0431DZTPJM> - it should be fixed.""}]"