8000 Added field to the RealmImport spec to replace environment variables within the realm import by stustison · Pull Request #26471 · keycloak/keycloak · GitHub
[go: up one dir, main page]

Skip to content

Added field to the RealmImport spec to replace environment variables within the realm import - #26471

Closed
stustison wants to merge 3 commits into
keycloak:mainfrom
stustison:import_env
Closed

stustison wants to merge 3 commits into
keycloak:mainfrom
stustison:import_env

Conversation

@stustison
@stustison stustison commented Jan 24, 2024 •
Copy link
Copy Markdown
Contributor

Closes #26470

@stustison
stustison requested review from a team as code owners January 24, 2024 19:51
@shawkins shawkins left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. Two thoughts - it will need some kind of test, and a Boolean would be preferred to allow for null as the default.

@stustison
Copy link
Copy Markdown
Contributor Author

Sure, I can add that in.

@stustison
stustison force-pushed the import_env branch 2 times, most recently from 5f41b9e to 4132146 Compare January 26, 2024 16:24
@shawkins shawkins left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, just one small consideration.

@shawkins shawkins Feb 5, 2024 •
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public Boolean isReplacePlaceholders() {
public Boolean getReplacePlaceholders() {

Thanks for the update - sorry I was out the latter part of the week and may have missed the change. One last thing - can this method and the references to it use the get convention instead of is?

@vmuzikar vmuzikar left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stustison Thank you for the PR!

I wonder what would be the full use case here. We add a supported field to the RealmImport CR but we don't officially support setting env vars on the Deployment/import Job (pod template is not considered supported). I presume the main problem this tries to solve is using Secrets for values inside the Realm representation. Would it rather make sense instead of introducing this switch to introduce a field to the RealmImport CR that would mount Secrets to env vars? We could even hide the fact it's env vars, the field could simply be called placeholders, it would take a name of the placeholder and a Secret/Key reference. Only if this field is specified, we'd internally set -Dkeycloak.migration.replace-placeholders=true.

WDYT @stustison @shawkins?

@shawkins
shawkins commented Feb 6, 2024
Copy link
Copy Markdown
Contributor

We add a supported field to the RealmImport CR but we don't officially support setting env vars on the Deployment/import Job (pod template is not considered supported)

You can use either the podtemplate or the additionalOptions - if you are okay with the naming convention.

Would it rather make sense instead of introducing this switch to introduce a field to the RealmImport CR that would mount Secrets to env vars?

I'm not familiar enough to know, and we don't seem to have many examples - but generally are these values specific to just import, or when should they be associated with the keycloak cr?

@vmuzikar
vmuzikar commented Feb 6, 2024 •
Copy link
Copy Markdown
Contributor

You can use either the podtemplate or the additionalOptions - if you are okay with the naming convention.

PodTemplate is an unsupported option. It feels weird to have a first-class citizen supported option replacePlaceholders that then requires usage of unsupported PodTemplate in order to use it.

additionalOptions is a supported option but not advocated to be used to set env vars. It's possible it won't even use env vars in the future but in any case it's an implementational detail hidden from user.

are these values specific to just import

AFAIK, yes. The mappings are specific to import. Hence I was suggesting to add a dedicated mappings field to the RealmImport CR.

Something like:

kind: KeycloakRealmImport
spec:
  keycloakCRName: example-kc
  placeholders:
    - name: foobar
      secret:
        name: mysecret
        key: mykey
  realm:
    ...

@shawkins
shawkins commented Feb 6, 2024
Copy link
Copy Markdown
Contributor

Something like:

Is this the only system property / env variables used by import? If not, it may make sense to instead keep a placeholder flag and a dedicated section for controlling the env.

@vmuzikar
vmuzikar commented Feb 6, 2024
Copy link
Copy Markdown
Contributor

Is this the only system property / env variables used by import?

AFAIK, yes. CC @pedroigor

@shawkins
Copy link
Copy Markdown
Contributor

@stustison after some more internal discussion, we'd like to move forward with the proposal above from @vmuzikar - unfortunately it adds a bit more complexity. The closest match for this logic we already have are the additionalOptions on the Keycloak CR with the ValueOrSecret class. Do you think you could adapt that for this purpose?

@stustison
Copy link
Copy Markdown
Contributor Author

@shawkins Yeah let me go back and take another look at what you guys are suggesting and I'll update the PR when I get to it.

@briantward
Copy link
Copy Markdown

@stustison I am interested in the progress of this request toward the solution proposed by @vmuzikar. I may have some time in the coming weeks to assist with this.

@stustison
Copy link
Copy Markdown
Contributor Author

@briantward Hey! Sorry I haven't had a chance to dig into this just yet I got busy on some other things. I'll be looking into this today and tomorrow.

@stustison
Copy link
Copy Markdown
Contributor Author

Added a first cut of the changes, but haven't updated any of the tests yet.

@stustison
Copy link
Copy Markdown
Contributor Author

@shawkins @vmuzikar I updated the tests and code (sorry it took so long for me to get around to it!) Let me know if this is more along the lines of what you had in mind.

@keycloak-github-bot keycloak-github-bot Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding t 10000 his comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@keycloak-github-bot
Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.adapter.servlet.BrokerLinkAndTokenExchangeTest#testExternalExchangeCreateNewUserUsingMappers

Keycloak CI - Base IT

java.lang.AssertionError: expected:<200> but was:<400>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
...

Report flaky test

org.keycloak.testsuite.adapter.servlet.BrokerLinkAndTokenExchangeTest#testExternalExchange

Keycloak CI - Base IT

java.lang.AssertionError: expected:<200> but was:<400>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
...

Report flaky test

org.keycloak.testsuite.adapter.servlet.BrokerLinkAndTokenExchangeTest#testAccountLinkNoTokenStore

Keycloak CI - Base IT

java.lang.AssertionError: Unexpected page. Current Page URL: http://localhost:8280/exchange-linking/link?realm=child&provider=parent-idp
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.keycloak.testsuite.adapter.servlet.BrokerLinkAndTokenExchangeTest.testAccountLinkNoTokenStore(BrokerLinkAndTokenExchangeTest.java:494)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...

Report flaky test

org.keycloak.testsuite.client.ClientTypesTest#testUpdateClientWithClientType

Keycloak CI - Base IT (4)

java.lang.AssertionError: Not expected to update client
	at org.junit.Assert.fail(Assert.java:89)
	at org.keycloak.testsuite.client.ClientTypesTest.testUpdateClientWithClientType(ClientTypesTest.java:112)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
...

Report flaky test

org.keycloak.testsuite.oauth.ClientTokenExchangeTest#testExchangeWithDynamicScopesEnabled

Keycloak CI - Base IT (6)

org.keycloak.common.VerificationException: Token not set
	at org.keycloak.TokenVerifier.parse(TokenVerifier.java:401)
	at org.keycloak.testsuite.oauth.ClientTokenExchangeTest.testExchange(ClientTokenExchangeTest.java:309)
	at org.keycloak.testsuite.oauth.ClientTokenExchangeTest.testExchangeWithDynamicScopesEnabled(ClientTokenExchangeTest.java:1021)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...

Report flaky test

org.keycloak.testsuite.oauth.ClientTokenExchangeTest#testClientExchange

Keycloak CI - Base IT (6)

java.lang.AssertionError: expected:<200> but was:<400>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
...

Report flaky test

org.keycloak.testsuite.oauth.ClientTokenExchangeTest#testIntrospectTokenAfterImpersonation

Keycloak CI - Base IT (6)

java.lang.AssertionError: expected:<200> but was:<400>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
...

Report flaky test

org.keycloak.testsuite.oauth.ClientTokenExchangeTest#testPublicClientNotAllowed

Keycloak CI - Base IT (6)

java.lang.AssertionError: expected:<403> but was:<400>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
...

Report flaky test

org.keycloak.testsuite.oauth.ClientTokenExchangeTest#testExchangeUsingServiceAccount

Keycloak CI - Base IT (6)

org.keycloak.common.VerificationException: Token not set
	at org.keycloak.TokenVerifier.parse(TokenVerifier.java:401)
	at org.keycloak.testsuite.oauth.ClientTokenExchangeTest.testExchangeUsingServiceAccount(ClientTokenExchangeTest.java:347)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
...

Report flaky test

org.keycloak.testsuite.oauth.ClientTokenExchangeTest#testImpersonation

Keycloak CI - Base IT (6)

java.lang.AssertionError: expected:<200> but was:<400>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
...

Report flaky test

org.keycloak.testsuite.oauth.ClientTokenExchangeTest#testImpersonationUsingPublicClient

Keycloak CI - Base IT (6)

java.lang.AssertionError: expected:<200> but was:<400>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
...

Report flaky test

@shawkins shawkins left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stustison this is looking good, thank you for taking this up again. Just a couple of comments.

Comment thread pom.xml Resolved
@shawkins shawkins left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stustison this looks great - can I have you undo the star import usage, then we should be good to merge this.

…within the realm import

Closes keycloak#26470

Signed-off-by: stustison <scott.tustison@gmail.com>
@stustison
Copy link
Copy Markdown
Contributor Author

@shawkins I removed the * imports, it should be good to go now.

shawkins
shawkins previously approved these changes Jun 28, 2024
@shawkins shawkins left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you @stustison for getting this past the finishline - it looks like others will appriciate this work.

@shawkins
Copy link
Copy Markdown
Contributor

I'm rerunning the failed tests - if they fail again we'll need to look into it more. There have been some intermittent flakiness with import, but the local here are failing as well which is unusual.

…within the realm import

Closes keycloak#26470

Signed-off-by: stustison <scott.tustison@gmail.com>
@stustison
Copy link
Copy Markdown
Contributor Author

@shawkins it should be passing now. I put the test back to the way it was before. The object sent to the function is a GenericKubernetesResource, which I can't really modify in any way. It was easier to just set it back to how it was previously when it was working.

@shawkins
Copy link
Copy Markdown
Contributor

Closing in favor of #31232

@shawkins shawkins closed this Jul 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a field to the RealmImportSpec to toggle replacing ENV variable placeholders

4 participants

0