🐛 Bug Report: PocketID gets indexed by web crawlers #386

New Issue

OVERLORD · 2026-02-04T19:22:23+03:00

OVERLORD commented

2026-02-04 19:22:23 +03:00

Originally created by @Etienne-bdt on GitHub (Jul 21, 2025).

Reproduction steps

Not really related to the app itself but, have a pocketid instance for some time and bad luck.
Google your domain
Tadah !

Expected behavior

It should not be indexed using robots.txt (iirc)

Actual Behavior

It gets crawled and indexed on google

Version and Environment

Caddy reverse proxy with the latest pocketid

Log Output

No response

Originally created by @Etienne-bdt on GitHub (Jul 21, 2025). ### Reproduction steps Not really related to the app itself but, have a pocketid instance for some time and bad luck. Google your domain Tadah ! ### Expected behavior It should not be indexed using robots.txt (iirc) ### Actual Behavior It gets crawled and indexed on google ### Version and Environment Caddy reverse proxy with the latest pocketid ### Log Output _No response_

OVERLORD closed this issue

2026-02-04 19:22:23 +03:00

OVERLORD commented

2026-02-04 19:22:38 +03:00

@bluewalk commented on GitHub (Jul 21, 2025):

Don't really think this is a bug though.

I've added a route block to Caddyfile for robots.txt, just to be sure it wouldn't get indexed.

route /robots.txt {
   header Content-Type "text/plain; charset=utf-8"
   respond <<EOT
User-agent: *
Disallow: /
EOT
}

@bluewalk commented on GitHub (Jul 21, 2025): Don't really think this is a bug though. I've added a route block to Caddyfile for robots.txt, just to be sure it wouldn't get indexed. ``` route /robots.txt { header Content-Type "text/plain; charset=utf-8" respond <<EOT User-agent: * Disallow: / EOT } ```

OVERLORD commented

2026-02-04 19:22:54 +03:00

@Etienne-bdt commented on GitHub (Jul 21, 2025):

Yeah, it might not really qualify as a bug, didn't know what to put in there
Thing is I never served robots.txt and never got indexed on my other services, but thanks I'll give your solution a try !

@Etienne-bdt commented on GitHub (Jul 21, 2025): Yeah, it might not really qualify as a bug, didn't know what to put in there Thing is I never served robots.txt and never got indexed on my other services, but thanks I'll give your solution a try !

OVERLORD commented

2026-02-04 19:23:00 +03:00

@ElioDiNino commented on GitHub (Jul 22, 2025):

It would be nice if there was an environment variable for whether to serve robots.txt or, alternatively, serve a default with the option to override. Most other self-hosted services block indexing by default.

@ElioDiNino commented on GitHub (Jul 22, 2025): It would be nice if there was an environment variable for whether to serve robots.txt or, alternatively, serve a default with the option to override. Most other self-hosted services block indexing by default.

OVERLORD commented

2026-02-04 19:23:04 +03:00

@stonith404 commented on GitHub (Jul 22, 2025):

Pocket ID shouldn't be indexed by crawlers. Are you sure your instance gets indexed?

https://github.com/pocket-id/pocket-id/blob/main/frontend/src/app.html#L7

@stonith404 commented on GitHub (Jul 22, 2025): Pocket ID shouldn't be indexed by crawlers. Are you sure your instance gets indexed? https://github.com/pocket-id/pocket-id/blob/main/frontend/src/app.html#L7

OVERLORD commented

2026-02-04 19:23:13 +03:00

@ElioDiNino commented on GitHub (Jul 22, 2025):

Ah yeah, the noindex meta tag should be enough to prevent indexing (although it doesn't stop Google and others from crawling). Not sure how @Etienne-bdt had his instance shown on Google 🧐

@ElioDiNino commented on GitHub (Jul 22, 2025): Ah yeah, the `noindex` meta tag should be enough to prevent indexing (although it doesn't stop Google and others from crawling). Not sure how @Etienne-bdt had his instance shown on Google 🧐

OVERLORD commented

2026-02-04 19:23:22 +03:00

@Etienne-bdt commented on GitHub (Jul 22, 2025):

Right, I just saw the noindex tag.
I'm positive it got indexed however.

Google doesn't provide a lot of info aside that they indexed the website...

@Etienne-bdt commented on GitHub (Jul 22, 2025): Right, I just saw the `noindex` tag. I'm positive it got indexed however. <img width="1080" height="257" alt="Image" src="https://github.com/user-attachments/assets/1698f304-da93-459d-8988-cbea54871a32" /> Google doesn't provide a lot of info aside that they indexed the website...

OVERLORD commented

2026-02-04 19:23:47 +03:00

@ItalyPaleAle commented on GitHub (Jul 30, 2025):

I guess we could still create a static robots.txt to include in our app... Wouldn't hurt, right?

@ItalyPaleAle commented on GitHub (Jul 30, 2025): I guess we could still create a static `robots.txt` to include in our app... Wouldn't hurt, right?

OVERLORD commented

2026-02-04 19:23:49 +03:00

@Etienne-bdt commented on GitHub (Jul 30, 2025):

I added my domain to the google search console and got "some" insight. According to the console indexation and exploration are both allowed.
If it's alright I'll open a PR to add the robots.txt with both disallow and no index directives.

@Etienne-bdt commented on GitHub (Jul 30, 2025): I added my domain to the google search console and got "some" insight. According to the console indexation and exploration are both allowed. If it's alright I'll open a PR to add the robots.txt with both disallow and no index directives.

OVERLORD commented

2026-02-04 19:23:56 +03:00

@kmendell commented on GitHub (Aug 2, 2025):

Added in https://github.com/pocket-id/pocket-id/pull/806

@kmendell commented on GitHub (Aug 2, 2025): Added in https://github.com/pocket-id/pocket-id/pull/806

OVERLORD referenced this issue

2026-02-04 20:51:51 +03:00

[PR #386] [MERGED] chore(translations): update translations via Crowdin #750

Sign in to join this conversation.