Migrate the SSOTest by msdaly200 · Pull Request #48367 · keycloak/keycloak

msdaly200 · 2026-04-22T12:47:43Z

Migrate the SSOTest

keycloak-github-bot · 2026-04-22T13:47:16Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.model.singleUseObject.SingleUseObjectModelTest#testCluster

Keycloak CI - Store Model Tests

java.lang.AssertionError: 
threads didn't terminate in time: [main (RUNNABLE):
	at java.management@25.0.2/sun.management.ThreadImpl.dumpThreads0(Native Method)
	at java.management@25.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:505)
	at java.management@25.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:493)
...

Report flaky test

org.keycloak.testsuite.model.user.UserModelTest#testAddRemoveUserConcurrent

Keycloak CI - Store Model Tests

java.lang.NullPointerException: Cannot invoke "org.keycloak.models.KeycloakSessionFactory.create()" because "factory" is null
	at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:463)
	at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:447)
	at org.keycloak.testsuite.model.KeycloakModelTest.inComittedTransaction(KeycloakModelTest.java:590)
	at org.keycloak.testsuite.model.KeycloakModelTest.inComittedTransaction(KeycloakModelTest.java:578)
...

Report flaky test

keycloak-github-bot

Unreported flaky test detected, please review

rmartinc

Thanks @msdaly200! Please remove the old test, this way we can check the differences. Review is difficult without differences. 😄

Remember that, when you move a database test, you need to add the annotation @DatabaseTest to the test. You can also add to specific tests if you don't want all the class to be executed in the database flow, this way only a the annotated tests will be there. Your test should be also executed in the DB jobs (postgresql, mariadb, oracle,...).

keycloak-github-bot

Unreported flaky test detected, please review

keycloak-github-bot · 2026-04-23T17:53:13Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.authz.PolicyEvaluationTest#

Keycloak CI - Base IT (3)

org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...

org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...

org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...

Report flaky test

vaceksimon

This is a pretty big PR with lots of changes. Please next time try to reduce the AI generated noise, at least for the comments. Thank you for the contribution @msdaly200

msdaly200 · 2026-04-27T08:21:54Z

Thanks @vaceksimon and @rmartinc for all your comments, much appreciated.

msdaly200 · 2026-04-27T09:22:11Z

@vaceksimon @rmartinc Assuming testsuite/integration-arquillian/tests/base/testsuites/database-suite needs updated too...ie ssotest removed?

keycloak-github-bot · 2026-04-27T12:30:44Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#salesPostEmptyConsumerPostURL

Keycloak CI - Adapter IT Strict Cookies

org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
	at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
...

Report flaky test

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#salesPostSigTest

Keycloak CI - Adapter IT Strict Cookies

org.awaitility.core.ConditionTimeoutException: Assertion condition defined as a Lambda expression in org.keycloak.testsuite.util.URLAssert URL expected to begin with: https://localhost:8543/auth/realms/demo/login-action ; actual URL: https://localhost:8643/sales-post-sig/ ==> expected: <true> but was: <false> within 10 seconds.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:119)
	at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:31)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1160)
...

Report flaky test

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#disabledClientTest

Keycloak CI - Adapter IT Strict Cookies

org.openqa.selenium.TimeoutException: 
Expected condition failed: waiting for wrapped: text ('Login requester not enabled') to be present in text in element found by By.xpath: //body (tried for 5 second(s) with 500 milliseconds interval)
Build info: version: '4.28.1', revision: '73f5ad48a2'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.17.0-1010-azure', java.version: '21.0.10'
Driver info: org.openqa.selenium.firefox.FirefoxDriver_ByGraphene
...

Report flaky test

keycloak-github-bot

Unreported flaky test detected, please review

rmartinc

Thanks @msdaly200! Just three minor things and I think that this is OK. Let's see if we don't forget to remove/add the test somewhere. 😄

vaceksimon

Found a few more things. I think we'll be finished with the next one. Thanks @msdaly200

vaceksimon · 2026-04-28T10:40:06Z

+            // Poll both password update events (order may vary due to identical timestamps)
+            EventRepresentation event1 = events.poll();
+            EventRepresentation event2 = events.poll();
+
+            EventRepresentation updatePasswordEvent;
+            EventRepresentation updateCredentialEvent;
+
+            if (EventType.UPDATE_PASSWORD == EventType.valueOf(event1.getType())) {
+                updatePasswordEvent = event1;
+                updateCredentialEvent = event2;
+            } else {
+                updatePasswordEvent = event2;
+                updateCredentialEvent = event1;
+            }
+
+            EventAssertion.assertSuccess(updatePasswordEvent)
+                    .type(EventType.UPDATE_PASSWORD)
+                    .details(Details.CREDENTIAL_TYPE, PasswordCredentialModel.TYPE);
+
+            EventAssertion.assertSuccess(updateCredentialEvent)
+                    .type(EventType.UPDATE_CREDENTIAL)
+                    .details(Details.CREDENTIAL_TYPE, PasswordCredentialModel.TYPE);


I think what the comment states is wrong

// Poll both password update events (order may vary due to identical timestamps)

Perhaps adding the driver.waiting().waitForOAuthCallback() call solved the issue. I tried running this version and the events were asserted in the correct order each time.

Suggested change

// Poll both password update events (order may vary due to identical timestamps)

EventRepresentation event1 = events.poll();

EventRepresentation event2 = events.poll();

EventRepresentation updatePasswordEvent;

EventRepresentation updateCredentialEvent;

if (EventType.UPDATE_PASSWORD == EventType.valueOf(event1.getType())) {

updatePasswordEvent = event1;

updateCredentialEvent = event2;

} else {

updatePasswordEvent = event2;

updateCredentialEvent = event1;

}

EventAssertion.assertSuccess(updatePasswordEvent)

.type(EventType.UPDATE_PASSWORD)

.details(Details.CREDENTIAL_TYPE, PasswordCredentialModel.TYPE);

EventAssertion.assertSuccess(updateCredentialEvent)

.type(EventType.UPDATE_CREDENTIAL)

.details(Details.CREDENTIAL_TYPE, PasswordCredentialModel.TYPE);

EventAssertion.assertSuccess(events.poll())

.type(EventType.UPDATE_PASSWORD)

.details(Details.CREDENTIAL_TYPE, PasswordCredentialModel.TYPE);

EventAssertion.assertSuccess(events.poll())

.type(EventType.UPDATE_CREDENTIAL)

.details(Details.CREDENTIAL_TYPE, PasswordCredentialModel.TYPE);

Thanks Simon for all the comments - I added this because it was failing for the MySQL build, so locally this works fine - events always come back in the expected way, but not for MySQL.

Signed-off-by: Marie Daly <marie.daly1@ibm.com>

msdaly200 · 2026-04-29T11:42:51Z

Hi @vaceksimon and @rmartinc again thanks for your hard work in reviewing this file. I added more commits again, from a different branch in an attempt to keep git history. So the main code hasn't changed. Can you review/approve when you get a chance?

vaceksimon

Thanks for the great work @msdaly200!

rmartinc

Thanks a lot @msdaly200! LGTM too!

msdaly200 requested a review from a team as a code owner April 22, 2026 12:47

vaceksimon self-requested a review April 22, 2026 12:56

keycloak-github-bot Bot added the flaky-test label Apr 22, 2026

keycloak-github-bot Bot reviewed Apr 22, 2026

View reviewed changes

rmartinc requested changes Apr 22, 2026

View reviewed changes

msdaly200 requested review from a team as code owners April 22, 2026 14:16

keycloak-github-bot Bot added team/cloud-native team/core-clients team/core-iam labels Apr 22, 2026

msdaly200 force-pushed the 48153 branch from 93f65b6 to 951cb81 Compare April 23, 2026 09:43

msdaly200 commented Apr 23, 2026

View reviewed changes

Comment thread tests/base/src/test/java/org/keycloak/tests/forms/SSOTest.java Resolved

msdaly200 force-pushed the 48153 branch from 32b8218 to e5d2d3d Compare April 23, 2026 16:40

keycloak-github-bot Bot reviewed Apr 23, 2026

View reviewed changes

msdaly200 mentioned this pull request Apr 24, 2026

Flaky test: org.keycloak.testsuite.authz.PolicyEvaluationTest# #48446

Closed

msdaly200 force-pushed the 48153 branch 2 times, most recently from 9b1dcc5 to 93f65b6 Compare April 24, 2026 10:27

vaceksimon suggested changes Apr 24, 2026

View reviewed changes

msdaly200 force-pushed the 48153 branch from 93f65b6 to ce6f0f9 Compare April 24, 2026 16:26

keycloak-github-bot Bot mentioned this pull request Apr 24, 2026

Flaky test: org.keycloak.testsuite.model.session.UserSessionPersisterProviderTest#testExpirationDistribution #47834

Closed

keycloak-github-bot Bot mentioned this pull request Apr 24, 2026

Flaky test: org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode #47421

Closed

msdaly200 force-pushed the 48153 branch from 2d94c94 to de440d2 Compare April 27, 2026 08:15

msdaly200 force-pushed the 48153 branch from ae382f3 to ca56d93 Compare April 27, 2026 08:28

keycloak-github-bot Bot mentioned this pull request Apr 27, 2026

Flaky test: org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionsAtRandomNode #48230

Closed

msdaly200 force-pushed the 48153 branch 2 times, most recently from ece5fac to 7f79a5f Compare April 27, 2026 12:29

keycloak-github-bot Bot reviewed Apr 27, 2026

View reviewed changes

msdaly200 force-pushed the 48153 branch 3 times, most recently from 089fa01 to 01936a0 Compare April 28, 2026 07:35

rmartinc requested changes Apr 28, 2026

View reviewed changes

Comment thread tests/base/src/test/java/org/keycloak/tests/forms/SSOTest.java Resolved

Comment thread tests/base/src/test/java/org/keycloak/tests/forms/SSOTest.java Resolved

Comment thread tests/base/src/test/java/org/keycloak/tests/forms/SSOTest.java Resolved

vaceksimon suggested changes Apr 28, 2026

View reviewed changes

msdaly200 force-pushed the 48153 branch 3 times, most recently from 6cd0845 to 1da5ea6 Compare April 29, 2026 09:26

msdaly200 added 3 commits April 29, 2026 10:31

Closes keycloak#48153, migrated SSOTest

9de6f72

Signed-off-by: Marie Daly <marie.daly1@ibm.com>

Closes keycloak#48153, updated following PR reviews

0d4e74e

Signed-off-by: Marie Daly <marie.daly1@ibm.com>

Closes keycloak#48153, updated associated test suite files

7126480

Signed-off-by: Marie Daly <marie.daly1@ibm.com>

msdaly200 force-pushed the 48153 branch from b038bb3 to 7126480 Compare April 29, 2026 09:31

vaceksimon approved these changes Apr 29, 2026

View reviewed changes

rmartinc approved these changes Apr 29, 2026

View reviewed changes

ahus1 approved these changes Apr 29, 2026

View reviewed changes

ahus1 merged commit 9f2ea27 into keycloak:main Apr 29, 2026
85 checks passed

msdaly200 deleted the 48153 branch April 30, 2026 10:08

Conversation

Uh oh!

Unreported flaky test detected

org.keycloak.testsuite.model.singleUseObject.SingleUseObjectModelTest#testCluster

org.keycloak.testsuite.model.user.UserModelTest#testAddRemoveUserConcurrent

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Unreported flaky test detected

org.keycloak.testsuite.authz.PolicyEvaluationTest#

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Unreported flaky test detected

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#salesPostEmptyConsumerPostURL

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#salesPostSigTest

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#disabledClientTest

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants