8000 Fix oid4vc tests by wistefan · Pull Request #29209 · keycloak/keycloak · GitHub
[go: up one dir, main page]
Skip to content

Fix oid4vc tests - #29209

Merged
mposolda merged 2 commits into
keycloak:mainfrom
wistefan:fix-oid4vc-tests
May 2, 2024
Merged

Fix oid4vc tests#29209
mposolda merged 2 commits into
keycloak:mainfrom
wistefan:fix-oid4vc-tests

Conversation

@wistefan
@wistefan wistefan commented May 2, 2024
Copy link
Copy Markdown
Contributor

Fixes the flakyiness of the OID4VC tests. The OID4VCGrantFeatureTest introduces a side-effect, that fails the following OID4VCI tests(thus the succeed if only the failed once are restarted).
Since the Feature test disabled an by-default disabled (experimental)feature and the test-framework enables all features that where disabled-by-annotation after the test, the OID4VCI-Feature was set to enabled in the testcontext. However, since the the following tests require a restart, they are disabled in the newley started instance, but enabled in the test-context.
The PR sets the feature state in the After-Test to the value set in the current profile or in the defaults(e.g. all EXPERIMENTAL will stay disabled).

closes #28982
closes #28983
closes #28984
closes #28985
closes #28986
closes #28987
closes #28988
closes #28989
closes #28990
closes #28991
closes #28992
closes #28993
closes #28994
closes #28995
closes #28996

wistefan added 2 commits May 2, 2024 11:14
@wistefan
only enable/disable features that should
2092848 
Signed-off-by: Stefan Wiedemann <wistefan@googlemail.com>
@wistefan
use default profile if nothing is set
c737bb0 
Signed-off-by: Stefan Wiedemann <wistefan@googlemail.com>
@wistefan
wistefan marked this pull request as ready for review May 2, 2024 09:48
@wistefan
wistefan requested review from a team as code owners May 2, 2024 09:48
@mposolda mposolda self-assigned this May 2, 2024
mposolda
mposolda approved these changes May 2, 2024
View reviewed changes
@mposolda mposolda left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wistefan Thanks!

@mposolda
mposolda enabled auto-merge (squash) May 2, 2024 10:41
@mposolda
mposolda merged commit 3e16af8 into keycloak:main May 2, 2024
keycloak-github-bot[bot]
keycloak-github-bot Bot reviewed May 2, 2024
View reviewed changes
@keycloak-github-bot keycloak-github-bot Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@keycloak-github-bot
keycloak-github-bot Bot commented May 2, 2024
Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.adapter.servlet.BrokerLinkAndTokenExchangeTest#testExternalExchangeCreateNewUserUsingMappers

Keycloak CI - Base IT

java.lang.AssertionError: expected:<200> but was:<400>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
...

Report flaky test

org.keycloak.testsuite.adapter.servlet.BrokerLinkAndTokenExchangeTest#testExternalExchange

Keycloak CI - Base IT

java.lang.AssertionError: expected:<200> but was:<400>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
...

Report flaky test

org.keycloak.testsuite.adapter.servlet.BrokerLinkAndTokenExchangeTest#testAccountLinkNoTokenStore

Keycloak CI - Base IT

java.lang.AssertionError: Unexpected page. Current Page URL: http://localhost:8280/exchange-linking/link?realm=child&provider=parent-idp
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.keycloak.testsuite.adapter.servlet.BrokerLinkAndTokenExchangeTest.testAccountLinkNoTokenStore(BrokerLinkAndTokenExchangeTest.java:494)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...

Report flaky test

org.keycloak.testsuite.client.ClientTypesTest#testUpdateClientWithClientType

Keycloak CI - Base IT (4)

java.lang.AssertionError: Not expected to update client
	at org.junit.Assert.fail(Assert.java:89)
	at org.keycloak.testsuite.client.ClientTypesTest.testUpdateClientWithClientType(ClientTypesTest.java:112)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
...

Report flaky test

org.keycloak.testsuite.oauth.ClientTokenExchangeTest#testExchangeWithDynamicScopesEnabled

Keycloak CI - Base IT (6)

org.keycloak.common.VerificationException: Token not set
	at org.keycloak.TokenVerifier.parse(TokenVerifier.java:401)
	at org.keycloak.testsuite.oauth.ClientTokenExchangeTest.testExchange(ClientTokenExchangeTest.java:309)
	at org.keycloak.testsuite.oauth.ClientTokenExchangeTest.testExchangeWithDynamicScopesEnabled(ClientTokenExchangeTest.java:1021)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...

Report flaky test

org.keycloak.testsuite.oauth.ClientTokenExchangeTest#testClientExchange

Keycloak CI - Base IT (6)

java.lang.AssertionError: expected:<200> but was:<400>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
...

Report flaky test

org.keycloak.testsuite.oauth.ClientTokenExchangeTest#testIntrospectTokenAfterImpersonation

Keycloak CI - Base IT (6)

java.lang.AssertionError: expected:<200> but was:<400>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
...

Report flaky test

org.keycloak.testsuite.oauth.ClientTokenExchangeTest#testPublicClientNotAllowed

Keycloak CI - Base IT (6)

java.lang.AssertionError: expected:<403> but was:<400>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
...

Report flaky test

org.keycloak.testsuite.oauth.ClientTokenExchangeTest#testExchangeUsingServiceAccount

Keycloak CI - Base IT (6)

org.keycloak.common.VerificationException: Token not set
	at org.keycloak.TokenVerifier.parse(TokenVerifier.java:401)
	at org.keycloak.testsuite.oauth.ClientTokenExchangeTest.testExchangeUsingServiceAccount(ClientTokenExchangeTest.java:347)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
...

Report flaky test

org.keycloak.testsuite.oauth.ClientTokenExchangeTest#testImpersonation

Keycloak CI - Base IT (6)

java.lang.AssertionError: expected:<200> but was:<400>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
...

Report flaky test

org.keycloak.testsuite.oauth.ClientTokenExchangeTest#testImpersonationUsingPublicClient

Keycloak CI - Base IT (6)

java.lang.AssertionError: expected:<200> but was:<400>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.a
8000
ssertEquals(Assert.java:633)
...

Report flaky test

robson90 pushed a commit to robson90/keycloak that referenced this pull request May 2, 2024
@wistefan @robson90
Fix oid4vc tests (keycloak#29209)
3181f0c 
closes keycloak#28982
closes keycloak#28983
closes keycloak#28984
closes keycloak#28985
closes keycloak#28986
closes keycloak#28987
closes keycloak#28988
closes keycloak#28989
closes keycloak#28990
closes keycloak#28991
closes keycloak#28992
closes keycloak#28993
closes keycloak#28994
closes keycloak#28995
closes keycloak#28996

* only enable/disable features that should

Signed-off-by: Stefan Wiedemann <wistefan@googlemail.com>

* use default profile if nothing is set

Signed-off-by: Stefan Wiedemann <wistefan@googlemail.com>

---------

Signed-off-by: Stefan Wiedemann <wistefan@googlemail.com>
Signed-off-by: Robin Meese <39960884+robson90@users.noreply.github.com>
@graziang
graziang commented May 9, 2024 •
edited
Loading
Copy link
Copy Markdown
Contributor

This introduces an issue when the feature is enabled on the class and then disabled on a method, tests subsequent to the one with @DisableFeature have the feature disabled like #29253

@wistefan
wistefan deleted the fix-oid4vc-tests branch June 4, 2024 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mposolda mposolda mposolda approved these changes
+1 more reviewer
@keycloak-github-bot keycloak-github-bot[bot] keycloak-github-bot[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@mposolda mposolda

Labels

flaky-test team/cloud-native team/core-clients team/core-iam

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Flaky test: org.keycloak.testsuite.oid4vc.issuance.signing.OID4VCIssuerEndpointTest#testGetCredentialOfferUnauthorized Flaky test: org.keycloak.testsuite.oid4vc.issuance.signing.OID4VCIssuerEndpointTest#testGetCredentialOfferUriUnauthorized Flaky test: org.keycloak.testsuite.oid4vc.issuance.signing.OID4VCIssuerEndpointTest#testRequestCredential Flaky test: org.keycloak.testsuite.oid4vc.issuance.signing.OID4VCIssuerEndpointTest#testGetCredentialOfferUriInvalidToken Flaky test: org.keycloak.testsuite.oid4vc.issuance.signing.OID4VCIssuerEndpointTest#testGetCredentialOfferUriUnsupportedCredential
Flaky test: org.keycloak.testsuite.oid4vc.issuance.signing.OID4VCIssuerEndpointTest#testRequestCredentialInvalidToken Flaky test: org.keycloak.testsuite.oid4vc.issuance.signing.OID4VCIssuerEndpointTest#testRequestCredentialUnauthorized Flaky test: org.keycloak.test 4B83 suite.oid4vc.issuance.signing.OID4VCIssuerEndpointTest#testCredentialIssuance Flaky test: org.keycloak.testsuite.oid4vc.issuance.signing.OID4VCIssuerEndpointTest#testGetCredentialOfferWithoutNonce Flaky test: org.keycloak.testsuite.oid4vc.issuance.signing.OID4VCIssuerEndpointTest#testGetCredentialOffer

3 participants

@wistefan @graziang @mposolda
0