Now that you understand the purpose of Self Service and you know where to find where, you want to get started. Depending on your situation, you either want to start from scratch with a new stream as indicated in the image below, or you want to consume the data on an existing stream.
Create an application in Self Service that represents your actual producer/consumer application (which is deployed somewhere)
Configure application principals for your application, so it can be authorized to produce to or consume from the stream. Note: you need to do this for every environment the application is running.
Request stream access, so your application can actually produce or consume.
Before creating a new stream (topic) most likely a schema must be discussed and finalized. This is currently outside the scope of Self Service. Once a schema has been developed and available for use, you can proceed with below steps to create a new stream in Self-Service.
For this purpose, you need to Create a private environment
Refer to Authorizing stream access
See here how to do that.
I already have set up an application principal, why do I need a keystore/truststore with my application?
This is required due to the way mTLS is set up.
The keystore/truststore is used for the “authentication” phase, the mTLS layer (provided by the JVM running Kafka Broker), which checks key validity period and whether the “Signed by” entry is in the truststore on the Broker side, and vice versa of course.
What you upload in Self Service will be used by the Broker itself for the “authorisation” phase, and uses a Quorum that holds ACLs to check whether the provided certificate in the SSL context, combined with the topic it requests access too, is valid or not.
In your installation you might want to avoid storing certificate files on a server, you can instead provide your certificate in string format which you can then plug into your SSL Configuration.
You would have to extract the entire chain from your keystore (similarly to how it was done at the time the application was registered on the Self Service UI).
|The following is assuming that you are using a bash terminal and not ZSH or another custom terminal to run the commands.|
Export the application .jks keystore into the PKCS12 format using
keytool, which generally comes with every JRE installation.
keytool -importkeystore \ -srckeystore srcKeystore.jks \ -srcstoretype jks \ -srcstorepass srcKeystorePassword \ -destkeystore destKeystore.p12 \ -deststoretype PKCS12 \ -deststorepass destpwd
Do the same for the application truststore
keytool -importkeystore \ -srckeystore srcTruststore.jks \ -srcstoretype jks \ -srcstorepass srcTruststorePassword \ -destkeystore destTruststore.p12 \ -deststoretype PKCS12 \ -deststorepass destpwd
You have now converted both the keystore and truststore in
PKCS12format, as an intermediate step.
Retrieve the application certificate and print as a PEM string, using the openssl command
openssl pkcs12 \ -in destKeystore.p12 \ -nokeys \ -passin pass:destpwd \ -passout pass:destpwd \ | grep -v -e '^\s' | grep -v '^\(Bag\|subject\|issuer\)' | sed -e 's/$/\\n/' | tr -d '\r\n'
This will print the keystore in PEM format on the console, in a single line with newline characters, so you can use it in your client configuration right away.
Do exactly the same, but now for the application truststore
openssl pkcs12 \ -in destTruststore.p12 \ -nokeys \ -passin pass:destpwd \ -passout pass:destpwd \ | grep -v -e '^\s' | grep -v '^\(Bag\|subject\|issuer\)' | sed -e 's/$/\\n/' | tr -d '\r\n'
Next, Export your private key as a PEM string
openssl pkcs12 \ -in destKeystore.p12 \ -nocerts \ -nodep \ -passin pass:destpwd \ -passout pass:destpwd \ | grep -v -e '^\s' | grep -v '^\(Bag\|Key\)' | sed -e 's/$/\\n/' | tr -d '\r\n'
This will print the private key in PEM format on the console, in a single line with newline characters
You have now prepared all PEM strings which you can use in the
SslConfigof your application.
Use the provided builder, to set
SslConfigand copy/paste the strings as generated with the commands above
SslConfig.newBuilder() .setKeystoreType(SslConfig.KeystoreType.PEM) .setKeystoreCertificateChain("----BEGIN CERTIFICATE----\n... ") .setKeystoreKey("----BEGIN PRIVATE KEY----\n... ") .setTruststoreType(SslConfig.TruststoreType.PEM) .setTruststoreCertificates("----BEGIN CERTIFICATE----\n... ") .build()
Short answer: YES
Long answer: Yes, but it takes a while to reach. There are various criteria to the use case, but the most important ones are these:
Available bandwidth / Networking speed for both Broker and producer/consumer
Available CPU and memory to both Broker and producer/consumer
Average message-size and/or batch-size
Choice between throughput or latency
Producer client acknowledgement. Does a client want to know:
a message has been sent to Broker;
a message has been received by Broker;
a message has been successfully submitted by Broker and its replicas.
Writing/Reading speeds of the disk(s) in the Broker
If you have no value for retention, you might be able to mount a RAMdisk on the Broker and send millions of messages of 10 bytes each second without any confirmation by Broker and a retention of 4 seconds. But in general you will want at least some form of confirmation that Broker has received your message, as well as some hours of retention to be able to mitigate downtime from any component. This means that HDD/SSD disks come into play, lowering transmission speeds by definition. Or it becomes REALLY expensive, real quick.