Repository navigation
Replies: 19 comments 99 replies
|
The first set of features are now available in the nightly build. I've updated the description. |
This is not tackling rolling upgrades. Rolling upgrades work as before: They are not officially supported. There is a good chance that rolling upgrades work when there is no change in the embedded Infinispan and no database changes. Still, rolling upgrades depend on a lot of different things, including Keycloak API and token changes (a token created by one Keycloak instance needs to work on the other Keycloak instance). We're working in different issues to improve that.
See above. The magic number issue might be due to Infinispan version changes.
This is not supported, and this is not on our roadmap. Instead, we'll work on a future version of Infinispan (including JGroups) and Keycloak that support rolling upgrades. As this is leaves open the question what is then changes with persistent sessions. Here a short overview:
|
|
Thanks for the clarifications. I think it would be great to add this to the documentation as a summary of "what this does":
|
|
Last week we had an ask-me-anything live session about persistent sessions. We even recorded it: https://www.youtube.com/live/yzdQqDIR0Y4 Please let us know if this is helpful, and continue any discussions here on GitHub. If you liked the format, please comment as well, and possibly suggest future topics. |
|
That is great help, and the format is good. Thanks! |
|
Update: Persistent user sessions are enabled by default in Keycloak's main branch. If everything goes well, they will be fully-supported and enabled-by-default for KC26. If you then choose to disable them, it will revert to the old functionality of storing sessions only in memory. |
|
Is Keycloak with It will use the database cache, so we hope it would run reasonable fast. If this is case, it would simplify the code of Keycloak a lot, and the data would always be consistently stored in Keycloak's database, and you need to worry less about Keycloak's JVM memory usage. Please comment here with your findings - possibly with your logins per second and number of concurrent user sessions. Thanks! |
|
We'd be happy to try this and give some feedback on this as we will care quite a bit if it's an issue in the future. Our stats are on average:
Do you mean the Hibernate entity cache, or hoping our DB has more RAM than ~ its data size? I imagine the latter, but for what it's worth Keycloak has never seemed that demanding to us on the RAM end of things. Sure it's has a relatively high base cost for small use-cases, but it seems to scale very well as the number of sessions increases. I assume we are into the fairly large category, and it seems to not even really want to use a mere 3GB of heap size (as per this comment), so if I had to choose between saving memory on the JVM or saving iops on Postgres, I'll choose the latter every day. Especially as IO capacity is usually a much scarcer/pricey resource than memory. Either way, your message notes:
And here I'm not sure how we could reasonably do that? Do you know how others try these? I mean we can run synthetic tests, potentially, but I assume you can do that already and are looking for real-world reports instead? And I'm not sure how to do that would massive impact on our users if it means more than 1 mass-dc event. |
I was hoping that the RAM of the database would be able to cache the pages for the active user sessions and the relevant indexes. We don't plan for Hibernate caching. The next iteration will have a bounded cache for sessions in Infinispan though, see #28318. There are some people with 50m+ sessions, and they struggle with the JVM memory this uses, so using the DB is an alternative. I agree IO is pricey, but the trade-off is different depending on the scenario. The current implementation will write to the database on each refresh of a token. A future version might batch those updates (which comes with tradeoffs) - I'll add this to the epic as another task. As you are asking on how to reasonably test this: Synthetic tests in test environments are the thing we're currently looking for, with the load tests that people created to match their production load. There is no one-size-fits-all load test, and we would need to know when we meet people's trade-offs between in-memory-only-and-gone-after-restart and store-in-database-which-is-more-expensive-but-ok-i-need-it. This will depend on business needs, available infrastructure and infrastructure costs. The discussion we're having now is also valuable, as it helps to steer our efforts into the right direction. While feedback during discussions and on nightly releases is usually fast, we have new major Keycloak releases about every three months which is comparatively slow. Once there is a release of Keycloak 25, this is then something you can properly upgrade to, and switch on-and-off in a production environment and we'll receive feedback. Any fixes to the code will need to wait for another major release except when they are small and they could then end up in a patch release. |
|
Hi @ahus1 . |
|
At the time, we were thinking it added more complexity for testing Keycloak and running it in a production environment to have two different options. Even with the "no cache" option, there would have been Infinispan for other areas, like for the authentication sessions, and for the work cache that is used to send out invalidations across a Keycloak cluster when entities change. With the now limited number of entries in the cache, IMHO the most observed problem of using too much memory or startup problems should now be gone. If you have scenarios explaining when not using a cache would be a good thing, please share them here to we can evaluate if the option would be re-introduced, or if there are other means to improve the situation you have in mind. |
|
Some production statistics :
count from ISPN database persistence (right now with the hacky
8 vCPU / 32 Go RAM, huge JVM around 20 Go (way too big, it has to be reviewed following recent guide and benchmark results) with most of users and user sessions in cache. CPU usage is low on both instances (KC & database), they are oversized for current trafic. NOTE: delay to get a refresh token has increased a lot between KC 20 and KC 22, from 40 ms to 220 ms with the same infrastructure, a little bit more sessions but not too much. We did not take time to investigate more on this. |
|
Yes, correct that's why I wasn't sure if I should have posted it here. Meanwhile, I was able to resolve my issue and I'm running succesfully keycloak 25 now :-). Thank you for the tips btw! |
Congrats to that! As a reference to future readers of this thread, please leave a hit on what the root cause was and how you solved it. It might also help the maintainers to improve the docs or simplify Keycloak in general so that other's don't need solve that problem in a future version of Keycloak any more. |
|
I misunderstood the upgrading guide with the --cache and --cache-config-file options. I thought that when removing them and adding the config file to the specific folder would automatically load in the file. Well, I was wrong about it :-) I needed to provide the variables as runtime variables in my container app. Maybe providing an example would have triggered me to process it correctly the first time :-) |
|
Thank you for sharing. It sounds to me like you directly edited the Is that the paragraph in the upgrading guide that confused you:
Maybe it should read "... so you need to remove them from the build phase and add them during runtime"? If yes, please provide a PR for https://github.com/keycloak/keycloak/blob/main/docs/documentation/upgrading/topics/changes/changes-25_0_0.adoc |
|
Yes, that's the paragraph that confuses me, because I thought by removing them and add it to the correct directory it would automatically load up the correct settings. Well ... it wasn't. A good idea! I'll create a PR! |
|
I've been tracking this discussion and its tendrils across github, trying to figure out what problem "persistent user sessions" solves to begin with.... Or to be honest, if it adressess a specific use case of mine: Will this feature allow users to remain offline for a "long" time (say 30 days), and be able to silently log back in again? (the goal being to make it as easy as possible for the user to continue using the application). From what I understand, this can be mimicked by misuing an offline token, but I also have been told that this is not what offline tokens are meant for (which is that they are meant for executing authorized tasks when the user is not online) and may introduce unneccessary security risks. Thinking about it, it seems that there is always additional risk involved by allowing this (since a token or some sort of password must always be stored on the, potentially vulnerable, client, but it seems to be acceptable for big apps like Gmail, and even secure apps like Protonmail. Thanks for any clarification! |
|
Great! The additional IO makes sense, but IMHO is a small price for the convenience it brings the user. Just so we are clear, this does mean that the client device holds on to a token of sorts? Will be keeping my eyes out for the next major release! :-D |
|
One addition to what @ahus1 said. In general, what you mentioned @Sjoerd82 can be solved also without this feature by increasing the session timeout to more than 30 days. However, the problem this is solving is, that today when you set timeout to more than 30 days and there is a Keycloak restart (all nodes fail or there is a version upgrade) in the meantime, all online sessions are lost because they live only in memory. When this feature is enabled, the sessions are stored also in the database, therefore the name The exciting part about this is this change is opening a few interesting doors for us because the new approach does not assume all sessions are present in memory and therefore we can have lower memory consumption and better scalability. |
The common case for these kind of long running user sessions is some web (browser) based application where the user closes the browser, and then opens it up the next day or week and wants to be logged in. For this to work, only the cookie Keycloak sets needs to be available. This will connect to user with their browser to an existing user session. The the tokens for the client(s) are then re-created from that. Depending on the client/application and possibly a specific interaction in the client, the client can ask the user to still re-authenticate if they haven't done so for the last X seconds. |
I totally agree @mhajas that the 2 improvements "keep sessions after shutdown & restart" and "lower memory consumption" (even with some trade-offs on DB IOs) will provide both a greater user experience and open interesting technical doors. Currently, with infinispan persistence, sometimes we have to truncate the session table because of unmarshalling problems due to infinispan upgrades, between majors version upgrades. |
|
The current nightly version already support upgrading. During an upgrade all entries in Infinispan will be cleared, an all entries will be reloaded on-demand from the database as needed. With this feature, the plan is to make this work in KC25, so upgrades to future minor and major versions should not loose online sessions. This is the same behavior you have today with offline sessions. |
|
@mhajas @ahus1 Really happy to see the progress on this. Per #27976 (comment) I'm curious if this will also (eventually) encompass User Sessions, Authentication Sessions, User Login Failures, and Single Use Objects. We have a similar experiment for all the "cached" objects using JPA via the |
|
This focuses on user sessions (online and offline) only, as there is the greatest pain, and this discussion should focus on these. Once this is GA, we'll consider the next step on this evolutionary journey. |
|
Thanks @ahus1 I'll open (or find) a separate discussion for the future issues. |
|
Instead of opening new issues now, I'd rather like you to hold back on new issues as there is already the one about seamless upgrades in #24655. Once this is closed, we can detail the next steps. For now, it would be great if you could provide feedback on the nightly builds or the referenced PRs above to get this ready for KC25, and to avoid too many scattered activities. |
|
I would be curious for which scenarios you know this would work, and for which other scenarios it wouldn't work. |
|
Hi All, We use Keycloak, and currently, version 23.0.7 is live. Our issue is that the session breaks when we modify a theme during a new deployment (unfortunately, every update for us involves theme modification), causing sessions to be cleared and users to be logged out, which is very inconvenient. We believe that #28271 would solve our problem. Therefore, our question is, when can we expect the deployment of this ticket? We tested the nightly, and it looks fine prefer not to deploy nightly builds into production Is there any plan that when you will deliver this development? :) Thank you in advance for your assistance and information :) |
|
Why don't you just test the nightly build in your test environment and see, if it solves your problem, instead of asking directly for production support? After your tests you can contribute and give valuable feedback to the team and all the other users here. |
|
We tested it and looks fine in our test environment, just we don't want to use nightly at prod. That's why i'm asked when we can expect KC25? Maybe my my wording was inaccurate, sorry |
|
Thanks for clarifying, sounds good! :) See the milestone https://github.com/keycloak/keycloak/milestone/47 for planned date. Don't (never) rely on it, see this as planned and hopefully to be available around this time. Things can change every time. |
When you re-deploy Keycloak with a new version of a theme, you should use rolling updates, where each instance of Keycloak is restarted one after the other. This is already supported in Keycloak for many versions. The one thing that Keycloak doesn't support is running different versions concurrently (like 24.0.1 and 24.0.2 at the same time, or 23.0.0 and 24.0.0). |
|
Will this change also allow (or at least make it easier) to display recently expired sessions for users in the account console? I am assuming that since Infinispan is a cache, it's a bit harder to query for all user sessions, including expired ones, and clearing the information from the cache as soon as possible is best. It would be a pretty good boon from a security standpoint to be able to review recent user sessions, similar to how Microsoft displays such information. |
|
@darius-m - no, this is not part of this effort. A current functionality is to enable events and store in Keycloak's database. I know a previous version of the account console was able to show some of the events, but the current doesn't allow that AFAIK. @ssilvert - I think I remember a Keycloak issue to get this functionality back, but I didn't find it. Can you please post an update here with the current plans / a link to the issue? |
|
+1. I'd be great if there is a REST API to retrieve user sessions to implement several important security features requested by our customers on the account console, such as showing the latest sessions, and if possible the user's location of each login session. |
|
@tu-pm In the account API thhere is already an endpoint to retrieve the current user's sessions, it's called However, expired sessions are not displayed. |
@ahus1 If the account REST API is still there for this then I guess it wouldn't hurt to add this function back in. If the REST API is missing then that probably means someone had a very good reason for getting rid of it. Maybe open a discussion on the dev list? |
|
I don't have much to add other than that this would be well received by our users. |
|
Works so far 👍, but didn't do comprehensive tests, yet. Is there any chance that the |
|
Hi @dasniko - while we want to make it configurable, we also want to understand the intent for it. I would hope 10000 would be a good default for a general use case, so I hope people wouldn't need to change it, as Keycloak sets it to 10000 once you enable persistent sessions and no other setting in the XML exists. Can you please share more context for your setup? |
|
@ahus1 I don't think we will be able to come up with a generic value that is applicable to all/most of deployments. It is the same as with the caching for other entities. By this number, you can control the ratio between memory consumption and database utilization. If someone wants to minimize queries to the database, they can increase the number; if they want to minimize memory utilization, they will lower the value. Also, 10000 may be a number that works well for medium deployments, however, for big deployments with millions of sessions this number may be basically useless and all queries will go to the database. We can try to to some investigation about how users set cache sizes, for example, for users. I thought tuning the cache sizes is necessary for bigger deployments but I may be wrong. |
|
Yes, I can imagine different scenarios where this could make sense. Still, I'd like to hear about the real-world scenario Nico has at hand for this. The more context I have, the more options I can consider to improve the situation. |
|
As a first thought I planned to set a sessions/clientSessions Business use-case : Considering this setup :
What is quite a challenge to test with keycloak-benchmark to have a good representativeness of reality and fine tuning before production is load test with a lot of active users + some extra load. It is easy to test a huge load with hundred of users making lots of login/logout/refresh but looking at the cache part, results might be a bit different (less memory footprint, more cache usage ... best performances). At least "he who can do more can do less" if the cluster supports high level of op/seconds. With a |
|
I'd think you would keep those that are active within an interval of 1-4 hours in the cache, but can try with larger caches and watch the cache hit ratio. I assume you also increased the size of the users cache already. While there are multiple entries for a user in the local user cache, there is exactly one entry for each user and client session. Reading sessions from the database should be relatively cheap. Writing to the database is then expensive when compared to a read operation, as that would create write ops in your database and will happen on every login / logout / refresh. |
|
The persistent sessions setup described here allows you to keep your user sessions with the database, something you need to deploy anyway. Having to deploy Redis adds some extra complexity to the setup IMHO, and in addition to that Redis changed their license recently. To continue the discussion, I need to understand what different behavior you want to see by adding Redis to the mix. Is it simpler to operate, will it offer higher availability, will it offer different behavior for the users of Keycloak? |
|
@ahus1 Redis is the industry standard, out of the box usually - users have to be able to use Redis cache in addition to other options.
|
Something should only change if it adds value. If we can run without it, it is simpler, which is better. I think we need to dig deeper to understand more if it is needed. And it needs to be weighted with the recent license changes around Redis.
I'd suggest to run tests first for the scenarios you want to apply it to. When you have low traffic (~50 logins and logouts per second), you'll see an average ~5 ms on a per login basis added for persistent sessions, where the overall login took ~50 ms due to password hashing, TLS connection etc. We tested it to up to 600 logins per second, which had a bit longer added latencies as we started to batch the updates to the database. If you choose to not batch them, you can still get below 60 ms on 50th percentile see below.
We see more and more Keycloak installations where the user sessions are kept alive for many months, for example in ecommerce installations. In those setups, these sessions are not considered a cache and you can "simply" login in again when your sessions is gone. Logging out people would result in reduced revenue as you'll see more abandoned shopping baskets. So having the chance to put those in a database, back them up or replicate them to a second location for a fail over together with the rest of your Keycloak database is a great improvement for such setups. I'd like to understand more about your setup to understand possible trade-offs. |
|
Hi @ahus1. Are the full results of your tests and benchmarks published somewhere? |
|
We're running those benchmarks daily in in the Keycloak Benchmark project and publish the results in the I recommend to run test in your own infrastructure to validate them. It would also provide us reassurance that we're on the right path. A summary will be available in the updated sizing guide together with the final release. The PR is still open: #32344 |
|
I was under the impression that when you enable the persistent sessions feature, you can switch the cache type to local. |
|
While this change handles user sessions, there are still other caches which still need: The distributed authentication cache which handles the session while users are logging in, and the work cache which handles invalidations of the local caches for realms and users across the cluster. We could now switch the caches for user and client sessions (both online and offline) to local caches or even remove them. But as they are empty, this won't change much in performance and efficiency. We might still do that in a future release though. |
|
A distributed cache is certainly useful, but if users on the balancer use sticky sessions, then it seems to me that you can only get by with a local cache. If some nodes are unavailable for some reason, the mechanism of permanent sessions from the database will work. I may be missing some points, and if you select KC_CACHE=local, some important part of the functionality will be lost (besides the distributed cache for authorization, the absence of which can be offset by sticky sessions). |
|
As @ahus1 said, there is still the invalidation cache. If an admin changes the realm config and his update request reaches node B, the cached realm data on node A needs to be invalidated and the current realm information needs to be re-read from the DB by node A. |
|
Getting this error intermittently every few hours after enabling persistent user sessions on 25.0.2 ERROR [org.hibernate.orm.jdbc.batch] (org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker) HHH100501: Exception executing batch [org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: update OFFLINE_USER_SESSION set BROKER_SESSION_ID=?,CREATED_ON=?,DATA=?,LAST_SESSION_REFRESH=?,REALM_ID=?,USER_ID=?,version=? where OFFLINE_FLAG=? and USER_SESSION_ID=? and version=?], SQL: update OFFLINE_USER_SESSION set BROKER_SESSION_ID=?,CREATED_ON=?,DATA=?,LAST_SESSION_REFRESH=?,REALM_ID=?,USER_ID=?,version=? where OFFLINE_FLAG=? and USER_SESSION_ID=? and version=? |
|
keycloak-jdbc-persistence-errors.log
Shared Postgres database, 2 data centers in kubernetes active/passive setup. 3 instances in each datacenter using built-in infinispan and dns_ping for discovery. Logs attached |
|
I'm also running into this issue. I have 3 Keycloak instances running on the same Postgres database. |
|
I had a look at the logs @muneebahmed provided. Those errors in the log might appear occasionally if a user session is updated concurrently. Keycloak is handling those error and is retrying, unfortunately we can't keep Hibernate or our transaction manager from logging those warnings and errors. As long as you don't see errors or warnings from |
|
I upgraded KC from 23.0.4 to 25.0.0, then to 25.0.6. I get similar errors with three replicas. @ahus1, did I understand correctly that they are not critical and can be ignored? |
|
@k-alanov - those exceptions are recoverable, still they should be rare as they would otherwise lead to performance problems. Those indicate a concurrent access to a single session, for example if a session is refreshed concurrently which - from an OpenID Connect standpoint - should never happen, still Keycloak handles that gracefully. So I'd say you should handle this as a performance problem and should suspect a misbehaving client. Looking at Keycloak's user logs should help you spot those. As listed here, successful events are logged on the debug level, and enabling those logging categories or changing the logging level with the listed SPI option might help you to find the behavior that triggers the situation. |
|
Hi, I tried to jump from v21.1.1 to 25.0.2 and there seems to be a blocker for this on my test env. I had a bit of trouble at first as updating my external ispn (managed by kubernetes operator) would redeploy and blank my sessions when updated to 15. I then tried to do the upgrade process without updating the ISPN version and staying on 14.0.9 ISPN. So, my sessions are created by 21.1.1 in ISPN 14 and get consumed from that same setup on the 25 startup. However, the conversion failed and I see an error message I traced this to https://github.com/keycloak/keycloak/pull/17525/files#diff-1445188baf4e2f8d90de96fc76d0423dd86425d2a0a9ede5c3e20d88c3f85077 on line 94 where the client id is set in v22+. So, likely 22 is the lower bound for upgrading sessions. I'll also note that the docs say the table I did test the persistence on 25 between total restarts and that works! tyty. Lastly, I'm trying to set some expectations for the stake holders about actual real life session persistence. Clearly the sessions are based on class definitions within KC - so, if we really want full persistence, we cant change the classes. What can we expect for this? Can I say that persistence will work between major releases? minor? |
|
The representation in Infinispan is considered an internal interface. The same is true for how Keycloak is storing sessions in the database. The only public API is Keycloak's REST API. While you can use the internal APIs, breaking changes are expected. For example in Keycloak 26, we're changing the serialization format to ProtoStream, which is actually Protocol Buffers. While this has the chance for you parse the data in a programming language agnostic way, it is an internal API. In another perspective: Once Keycloak stored the sessions in the database, I assume most people would stop using an external Infinispan unless they use it to for example replicate its contents to other sites. |
|
@sortafullstack FYI: The fix was released today in version 25.0.4 feel free to test it and let us know whether it works for you. https://www.keycloak.org/2024/08/keycloak-2504-released.html |
|
Hi @mhajas, I tried it again with nightly on friday but ended up looking at a |
|
@sortafullstack - if the problem re-appears, please provide the complete stack trace as it would help us to find the root cause of the problem. |
|
@ahus1, that was with the nightly branch. When I switched to 24.0.4, it worked correctly and my sessions were available after the upgrade. Tks for the quick work on fixing the client issue! |
|
Anybody have success with Session transition with pretty large sets? Ie 300k+. Having issues that arent obvious how to fix. I posted an issue here : #32994. |
|
Found the issue, nm. See issue for deets. |
|
We are testing KC 26 for our deployment. let me give you a brief idea about our infra and usage. at present we are using kc 24.03. we use AWS EKS with single site deployment. we have minimum 3 nodes and 3 pods (m6i.large). we use external ISPN with mariadb for session persistence. we have around 11 million sessions keys in ispn. daily we have around 200k sessions. We use custom TCP stack for jgroup discoveries (tcpping). we firefight different issues that aries daily but managing such number sessions is the most challenging part. in our current deployment we see a issue that is in admin panel sessions are not showing(from user-> session) even through it exists in remote ispn because cache is refreshing client login is working as it should. but due to this problem 2 issues we are facing,
so when we found out 26 is offering session persistence by default we were very excited. but during our initial testing we found few strange behavior.
so here is my question, if sessions are persisted in db shouldn't all the pod have the data always ? and shouldn't all sessions are visible from user menu in admin? or do have to do something to make this work? |
|
@ahmadsajid1989 - thank you for sharing this. I updated your question with a link break as all items where showing as a single list, where I assume it should have been two lists. Please verify if your question shows now correctly. Please provide a list of all parameters you are passing to Keycloak, plus the startup log. Also scale to a single node and let me know if that works as expected to rule out problems in the discovery. Note that even if you are running with persistent sessions, the discovery in Keycloak needs to work. I also need to understand if you are using an external Infinispan only with KC24, and stopped using it for KC26? |
|
thanks for the correction and pointing out the node discovery part @ahus1 , we will test and get back with the results. also im collecting all the params so i can share here. |
|
I'm trying to apply it. But I get a driver error when configuring the database. Because I have one set in the keycloak .conf for the rest of the data. But it has to go in environment variables and when I put it I get this ( 2024-11-13 12:48:01,939 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Could not start server in (production) mode 2024-11-13 12:48:01,939 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Could not get JDBC connection 2024-11-13 12:48:01,939 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: The handler does not support the given URL: jdbc:mysql://localhost:3306/mowiz23keycloak?useSSL=false&allowPublicKeyRetrieval=true 2024-11-13 12:48:01,939 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details, run the same command passing the '--verbose' option. You can also use '--help' to see details about using a particular command.) |
|
my Dockerfile is( ENV KEYCLOAK_ADMIN=admin WORKDIR /opt/keycloak ENTRYPOINT ["/opt/keycloak/bin/kc.sh", "build", "&&", "/opt/keycloak/bin/kc.sh", "start", "--http-enabled=true", "--hostnam> |
|
I have read all the documents related to HA and multi-site deployment, but I couldn't find a clear explanation for: "Why do I need an external Infinispan for multi-site deployment?" From what I understand, the default Keycloak already uses an internal Infinispan for caching. Restarting one instance should be safe since it will rebalance from the healthy ones, right? We have an active-active database setup with both data centers being writable (MariaDB). |
|
I was wondering the same, so this is for future reference: Please, also see https://www.keycloak.org/2024/12/storing-sessions-in-kc26.html (FAQ)
|
|
We are in a very old version of Keycloak (15.x) with around 4000 users and we are upgrading to 26.x. (2 Node Cluster). This instance is one of the [critical] moving parts of a large platform. While the upgrade went fine in QA, we notice that few of our OIDC clients randomly gets unauthorized error while calling the Token verification endpoint, and it goes away if we shutdown one of our nodes, indicating this might be an issue with session persistence. Below is our startup command:
Below is the startup log I also checked if database table Any pointers on troubleshooting this would be great. Thanks ! |
|
Thank you for reaching out. There might be multiple causes for this, and I don't think this is related persistent sessions.
|
|
Thanks a lot. I did check the logs of second node as you said, it did not create a cluster. I also see below logs in Keycloak UI. Below are the events in keycloak for same user session (Getting session not found for some requests which I believe the requests may be going to second node) Now I am a little bit confused. I thought, since user sessions are stored in database rather than Infinispan Cache, my understanding was second node using the same database would recognize the token - It looks like it doesn't unless otherwise configured as a cluster. Is this understanding correct? Also can you kindly point me to any documentation for setting up a cluster for bare metal / virtual machines if its available? Again, I apologize if my above question is wrong. |
|
It is correct that KC needs the cluster to work. See https://www.keycloak.org/server/caching for now to configure the clustering for caches - sorry for including this above. What kind of bare metal / VM are you using? Is MPING as a discovery available? This is what might happen: A node reads the session from the database when it don't know about the session yet. It will cache those items in memory. And if the other node modifies it, it needs to invalidate the cache. It will also cache a lot of realm and user related data, and that needs to be invalidated as well if it is modified on another node. Again the clustering of Keycloak takes care of that. We're making this a lot simpler for the upcoming 26.1 release due in January. No need for configuring multicast and others, see #34102 |
|
Thanks a lot for explaining. We are in Amazon EC2, so looks like we have to use NATIVE_S3_PING. Or can we use TCP/JDBC_PING ? Or can we hard code the node IP`s in Cache Config? Thanks for all the hard work going into upcoming 26.1 release. That certainly looks promising. Kudos !! |
|
Yes, We never provided a JDBC_PING stack in 26 and earlier, and the default Maybe it is an option for you to wait for 26.1 and until then run with just one node. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
One of the biggest pain points of Keycloak is currently losing sessions on restarts. There were countless complaints on this, including a lot of workarounds that are usually unsupported and come with some trade-offs.
The current approach for storing sessions has become a blocker for active/active multi-site support that we are currently pursuing and therefore we needed to rethink this approach. While brainstorming a new approach we came up with a conclusion that we should also include durable sessions as that feature was already on our radar.
This is now part of Keycloak 25, see https://www.keycloak.org/docs/latest/upgrading/index.html#persistent-user-sessions for more details on how to migrate. Please provide your feedback here (where it works, and were it should be enhanced).
There is also a blog post Keeping users logged in with Keycloak 25 summarizing the changes.
In the code branch for Keycloak 26 which we use for our nightly builds, this is now enabled-by-default, and fully supported. If you choose to disable the feature, Keycloak will revert to the old functionality of storing sessions only in memory.
Idea
The current behavior of Keycloak is that online sessions are stored only in Infinispan while offline sessions are also stored in the database. Therefore, the new feature's main idea is to store online sessions in the same table structure.
Plans
To follow this work proceed to the following GitHub epic: #28265
Highlights of the Keycloak 25 release
How to test
Download Keycloak 25 or later
Run Keycloak with the
persistent-user-sessionsfeature. Proceed to the following guide on how to enable/disable Keycloak features.Example command in development mode:
To confirm that online sessions are stored in the database check the database table
offline_user_sessioncontains rows withoffline_flagcolumn equal to0.All reactions