8000 Adding staleness concept to JDBC_PING by ahus1 · Pull Request #47971 · keycloak/keycloak · GitHub
[go: up one dir, main page]

Skip to content

Adding staleness concept to JDBC_PING - #47971

Merged
ahus1 merged 5 commits into
keycloak:mainfrom
ahus1:is-47970-jdbc-ping-staleness
Apr 17, 2026
Merged

ahus1 merged 5 commits into
keycloak:mainfrom
ahus1:is-47970-jdbc-ping-staleness

Conversation

@ahus1
@ahus1 ahus1 commented Apr 11, 2026 •
Copy link
Copy Markdown
Member

This improves Keylcoak's behavior in two ways:

  • When one of the partitions dies, its entries will become stale, and will eventually be ignored and deleted
  • When one of the partition changes, for example with a node joining or leaving, the view change will no longer review the entries of the other partition, thereby making the split brain detection more stable (where it would previously would detect it only after the other partition renewed their entries after 30-45 seconds)

Not covered here:

  • If a partition vanishes, and the other partition takes over, the caches of the remaining partition are not cleared. This might be tackled in a follow-up PR.

Closes #47970

@keycloak-github-bot
Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.model.singleUseObject.SingleUseObjectModelTest#testCluster

Keycloak CI - Store Model Tests

java.lang.AssertionError: 
threads didn't terminate in time: [main (RUNNABLE):
	at java.management@25.0.2/sun.management.ThreadImpl.dumpThreads0(Native Method)
	at java.management@25.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:505)
	at java.management@25.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:493)
...

Report flaky test

org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionsAtRandomNode

Keycloak CI - Store Model Tests

java.lang.AssertionError: Execution didn't complete: java.lang.RuntimeException: org.infinispan.commons.TimeoutException: ISPN000427: Timeout after 15 seconds waiting for acks ([node-14]). Id=3361, Topology Id=2
	at org.junit.Assert.fail(Assert.java:89)
	at org.keycloak.testsuite.model.KeycloakModelTest.inIndependentFactories(KeycloakModelTest.java:448)
	at org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest.testPersistenceMultipleNodesClientSessionsAtRandomNode(OfflineSessionPersistenceTest.java:216)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
...

Report flaky test

org.keycloak.testsuite.model.session.AuthenticationSessionTest#testConcurrentAuthenticationSessionsCreation

Keycloak CI - Store Model Tests

java.lang.AssertionError: 
threads didn't terminate in time: [main (RUNNABLE):
	at java.management@25.0.2/sun.management.ThreadImpl.dumpThreads0(Native Method)
	at java.management@25.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:505)
	at java.management@25.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:493)
...

Report flaky test

org.keycloak.testsuite.model.session.UserSessionProviderModelTest#testCreateUserSessionsParallel

Keycloak CI - Store Model Tests

java.lang.AssertionError: 
threads didn't terminate in time: [main (RUNNABLE):
	at java.management@25.0.2/sun.management.ThreadImpl.dumpThreads0(Native Method)
	at java.management@25.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:505)
	at java.management@25.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:493)
...

Report flaky test

@keycloak-github-bot keycloak-github-bot Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

Closes keycloak#47970

Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
@ahus1
ahus1 force-pushed the is-47970-jdbc-ping-staleness branch from abaae9a to ff49dbd Compare April 11, 2026 15:37
8000
@keycloak-github-bot keycloak-github-bot Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@keycloak-github-bot
Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.model.singleUseObject.SingleUseObjectModelTest#testCluster

Keycloak CI - Store Model Tests

java.lang.AssertionError: 
threads didn't terminate in time: [main (RUNNABLE):
	at java.management@25.0.2/sun.management.ThreadImpl.dumpThreads0(Native Method)
	at java.management@25.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:505)
	at java.management@25.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:493)
...

Report flaky test

org.keycloak.testsuite.model.singleUseObject.SingleUseObjectModelTest#testCluster

Keycloak CI - Store Model Tests

java.lang.AssertionError: 
threads didn't terminate in time: [main (RUNNABLE):
	at java.management@25.0.2/sun.management.ThreadImpl.dumpThreads0(Native Method)
	at java.management@25.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:505)
	at java.management@25.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:493)
...

Report flaky test

@ahus1
ahus1 requested a review from pruivo April 13, 2026 06:51
@ahus1
ahus1 commented Apr 13, 2026
Copy link
Copy Markdown
Member Author

@pruivo - can you please review this PR when you have the time? Thanks!

Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
@ahus1
ahus1 commented Apr 16, 2026
Copy link
Copy Markdown
Member Author

@ryanemerson - I've now added a test, and refined the logic a bit more. Please review when you have the time. Thanks!

@ahus1
ahus1 marked this pull request as ready for review April 16, 2026 11:43
@ahus1
ahus1 requested review from a team as code owners April 16, 2026 11:43
"coordinated_by varchar(200), " +
"PRIMARY KEY (address) )";
insert_single_sql="INSERT INTO jgroups values (?, ?, ?, ?, ?, ?, ?)";
select_all_pingdata_sql="SELECT address, name, ip, coord, coordinated_by, last_update FROM jgroups WHERE cluster=?";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this is only required by JdbcPing2Test, as we're initializing these values in the JGroupsConfigurator, therefore it would be better if we added this to the KEYCLOAK_JDBC_PING2_FOR_TESTING constructor instead of in the production code.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we move this to upstream, those SQLs need to be updated there as well. That's why I want to keep them here.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I would suggest we create the upstream PR before we forget this detail, it doesn't feel right to include this code in src/main just for the purpose of a reminder. As the class variables in JDBC_PING2 are not initialized in a constructor, it is done inline, we won't be able to copy and paste this code in its current form directly anyway.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having the table name hardcoded here doesn't seem right. Do we have 2 tables now? jgroups and jgroups_ping?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ryanemerson - I've moved it from a constructor to a an initializer, as this might make it more closely aligned with what's happening in the parent class. Let me know your thoughts.

About upstream: Not sure if this will land in upstream as an update to JDBC_PING2, or rather a new variant JDBC_PING3 with some more opinionated defaults, and some pruned functionality for better maintenance.
To validate this approach, I'd like to start in Keycloak's repository.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pruivo - The name jgroups is the one used JGROUPS_PING2, and as I was about to move it upstream eventually, I was aligning with that name (and it is hardcoded there as well).

The person who implemented it for Keycloak chose the name jgroups_table, and that's how we ended up with a different name.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've moved it from a constructor to a an initializer

-1 from me. Configure the new SQL queries in model/infinispan/src/test/resources/jdbc-h2.xml.

Not sure if this will land in upstream as an update to JDBC_PING2

This is a backwards-incompatible change. Remember, there are JGroups users outside Infinispan and Keycloak, and they most likely don't have Liquibase or any tool to update the table schema :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The person who implemented it for Keycloak chose the name jgroups_table

Not entirely true, JDBC_PING has JGROUPSPING for table name: https://github.com/belaban/JGroups/blob/master/src/org/jgroups/protocols/JDBC_PING.java#L57

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-1 from me. Configure the new SQL queries in model/infinispan/src/test/resources/jdbc-h2.xml.

+1 to adding here, I wasn't aware of this file before.

This is a backwards-incompatible change. Remember, there are JGroups users outside Infinispan and Keycloak, and they most likely don't have Liquibase or any tool to update the table schema :)

That's a good point, at this stage JDBC_PING3 is a more realistic upstream contribution.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not entirely true, JDBC_PING has JGROUPSPING for table name:

I've been looking at JDBC_PING2 which is using jgroups, see https://github.com/belaban/JGroups/blob/master/src/org/jgroups/protocols/JDBC_PING2.java#L56.

I've added it to the jdbc-h2.xml file as you both suggested.

s.setString(3, clusterName); // cluster name
s.setString(4, "127.0.0.1:0"); // ip = new IpAddress("localhost", 0).toString()
s.setBoolean(5, false); // coord
s.setLong(6, TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis())); // coord
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be Time.currentTimeMillis() instead of System.currentTimeMillis()?

Comment needs updating to reflect that it's the staleness timestamp "last_update".

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in next commit

@ahus1
ahus1 requested a review from ryanemerson April 16, 2026 16:10
@ahus1
ahus1 commented Apr 16, 2026
Copy link
Copy Markdown
Member Author

@ryanemerson - this should now be ready for a re-review. Please have a look when you have the time. Thanks!

Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
ahus1 added 2 commits April 17, 2026 09:27
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
@ryanemerson ryanemerson left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @ahus1

@ahus1
ahus1 enabled auto-merge (squash) April 17, 2026 12:31
@ahus1
ahus1 merged commit 1df4b4a into keycloak:main Apr 17, 2026
85 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JDBC_PING2 will stall when in a two-node-split-brain one node dies

4 participants

0