8000 Provide an HTTP 503 response when the thread pool rejected the request by ahus1 · Pull Request #21963 · keycloak/keycloak · GitHub
[go: up one dir, main page]
Skip to content

Provide an HTTP 503 response when the thread pool rejected the request - #21963

Merged
vmuzikar merged 1 commit into
keycloak:mainfrom
ahus1:is-21962-load-shedding-response
Jul 26, 2023
Merged

Provide an HTTP 503 response when the thread pool rejected the request#21963
vmuzikar merged 1 commit into
keycloak:mainfrom
ahus1:is-21962-load-shedding-response

Conversation

@ahus1
@ahus1 ahus1 commented Jul 25, 2023
Copy link
Copy Markdown
Member

This allows for limiting the queue of the thread pool to avoid overloading Keycloak.

Closes #21962

@ahus1 ahus1 self-assigned this Jul 25, 2023
@ghost ghost added the flaky-test label Jul 25, 2023
ghost
ghost reviewed Jul 25, 2023
View reviewed changes
@ghost ghost left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@ghost
ghost commented Jul 25, 2023
Copy link
Copy Markdown

Unreported flaky test detected

If the below flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.forms.LoginTest#loginDifferentUserAfterDisabledUserThrownOut

Keycloak CI - Forms IT (firefox)

java.lang.AssertionError: Expected LoginPage but was Keycloak Account Management (https://localhost:8543/auth/realms/test/account/#/)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.keycloak.testsuite.pages.AbstractPage.assertCurrent(AbstractPage.java:47)
	at org.keycloak.testsuite.pages.LoginPage.open(LoginPage.java:236)
...

Report flaky test

@ahus1
ahus1 marked this pull request as ready for review July 25, 2023 20:00
@ahus1
ahus1 requested review from a team as code owners July 25, 2023 20:00
vmuzikar
vmuzikar requested changes Jul 26, 2023
View reviewed changes
@vmuzikar vmuzikar left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @ahus1. This change makes sense to me, although I still think it'd make sense to have this logged (perhaps as a warning?) on top of the HTTP 503 response. That'd help to make it clear what's happening to a user who's not very familiar with the internals.

@ahus1
ahus1 force-pushed the is-21962-load-shedding-response branch from 091e0f5 to a170899 Compare July 26, 2023 09:36
@ahus1
Provide an HTTP 503 response when the thread pool rejected the request
5278d99 
This allows for limiting the queue of the thread pool to avoid overloading Keycloak.

Closes keycloak#21962
@ahus1
ahus1 force-pushed the is-21962-load-shedding-response branch from a170899 to 5278d99 Compare July 26, 2023 09:46
@ahus1
ahus1 commented Jul 26, 2023
Copy link
Copy Markdown
Member Author

@vmuzikar - logging a warning for each dropped request would be an anti-pattern as it would amplify the load on the logging system. In our load tests I expect this to drop thousands of messages in an overloaded situation.

I've updated this PR to log a warning when Keycloak starts dropping requests and then another warning once it continues processing with an approximation of the dropped requests.

The output is as follows:

2023-07-26 11:36:01,982 WARN [org.keycloak.quarkus.runtime.integration.web.QuarkusRequestFilter] (vert.x-eventloop-thread-0) Executor thread pool exhausted, starting to reject requests
2023-07-26 11:36:02,031 WARN [org.keycloak.quarkus.runtime.integration.web.QuarkusRequestFilter] (vert.x-eventloop-thread-0) Executor thread pool no longer exhausted, request processing continues after 4 discarded request(s)

We can tune this more once we see it is useful in our tests.

@ahus1
ahus1 requested a review from vmuzikar July 26, 2023 09:50
vmuzikar
vmuzikar approved these changes Jul 26, 2023
View reviewed changes
@vmuzikar vmuzikar left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ahus1 Thanks for the update.

Yeah, makes sense to aggregate the logs to just start and end of the pool exhaustion.

@vmuzikar
vmuzikar enabled auto-merge (squash) July 26, 2023 10:03
@ghost ghost mentioned this pull request Jul 26, 2023
Closed
@vmuzikar
vmuzikar merged commit a1e1fd8 into keycloak:main Jul 26, 2023
@sschu
sschu commented Aug 2, 2023
Copy link
Copy Markdown
Contributor

@ahus1 Would it be possible to expose the number of rejected requests and also the waiting queue length as metric?

@ahus1 ahus1 mentioned this pull request Aug 3, 2023
Closed
@ahus1
ahus1 commented Aug 7, 2023
Copy link
Copy Markdown
Member Author

@sschu - I've raised an issue here: quarkusio/quarkus#34998

I still need to reply there, as I think Vert.x is not exposing of all the metrics we would want to see.

When running with the OTEL agent, I saw the 503 in the responses as a counter, which was working for me.

The current queue length would be nice indeed. Maybe even nicer: A histogram of the queue length capturing the length of the queue when something was added, so we wouldn't be limited to the value which is polled every some seconds? But I think the chances to get that would be nearly zero.

@ahus1 ahus1 mentioned this pull request Sep 18, 2023
Closed
@ahus1
ahus1 deleted the is-21962-load-shedding-response branch September 21, 2023 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vmuzikar vmuzikar vmuzikar approved these changes
+1 more reviewer
Reviewers whose approvals may not affect merge requirements

Assignees

@ahus1 ahus1

Labels

flaky-test team/cloud-native

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Minimal load shedding support for incoming requests

3 participants

@ahus1 @sschu @vmuzikar
0