Gorse + ClickHouse Cloud Incompatibility

Problem

Gorse recommendation engine fails to connect to ClickHouse Cloud with EOF errors during protocol negotiation, even though TLS handshake succeeds.

Context / Trigger Conditions

• Gorse master/server pods show CrashLoopBackOff
• Logs show: failed to connect data database: EOF or transport failed to send a request to ClickHouse: EOF
• Using ClickHouse Cloud (*.clickhouse.cloud) as the GORSE_DATA_STORE
• TLS connection works when tested with openssl s_client
• Tried various connection parameters without success

Root Cause

Despite the clickhouse:// URL scheme suggesting native protocol, Gorse actually uses the HTTP interface on port 8123 (not the native protocol on port 9000/9440). ClickHouse Cloud's HTTP interface on port 8443 has compatibility issues with Gorse's driver.

Key facts:

• Gorse uses ClickHouse HTTP interface (port 8123), NOT native protocol (port 9000)
• ClickHouse Cloud port 8443: HTTP/HTTPS protocol
• ClickHouse Cloud port 9440: Native protocol with TLS
• Gorse only accepts: clickhouse://, mysql://, postgres://, mongodb://
• Gorse does NOT support https:// scheme (returns "unsupported data storage backend")
• Redis is only for cache store, NOT data store
• If you use port 9000 with in-cluster ClickHouse, you'll get: "Port 9000 is for clickhouse-client program. You must use port 8123 for HTTP."

Solution Options

Option 1: Use In-Cluster ClickHouse (Recommended)

Deploy ClickHouse Operator with an in-cluster instance for Gorse:

# Standard in-cluster connection works
GORSE_DATA_STORE: "clickhouse://user:password@clickhouse-host:8123/gorse"

Option 2: Use Different Database

Gorse supports MySQL and PostgreSQL as data stores:

GORSE_DATA_STORE: "mysql://user:password@host:3306/gorse"
# or
GORSE_DATA_STORE: "postgres://user:password@host:5432/gorse"

Option 3: Deploy ClickHouse Proxy

Set up a proxy (like chproxy) that translates between protocols.

Option 4: Fallback to Popular Videos

Keep Gorse disabled and let the API fallback to popularity-based recommendations:

{"videos": [...], "source": "popular"}

Verification

After choosing a solution, check Gorse master logs:

kubectl logs -l app=gorse-master -n gorse --tail=20

Should show successful connection instead of EOF errors.

Notes

• The TLS layer works fine - the issue is at the ClickHouse protocol level
• This may be fixed in future versions of the clickhouse-go driver
• In-cluster ClickHouse doesn't have this issue because it uses plain connection on port 8123