No URLs found

[exportio]

Number of found URL : 1
Number of links crawled : 1

python main.py --domain https://www.domain.com --output sitemap.xml --report

<urlset
      xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
      xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9
            http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">

</urlset>

[c4software]

Hi,

Interesting… I don't have the problem here. What is your python version ?

[c4software]

@samboustani Problem still present ?

[GovetaXV]

Same problem here.

[GovetaXV]

try this url: https://paperarchive.space/

[c4software]

@GovetaXV Hi,

Thanks for the link. Unfortunately the current version of python-sitemap doesn't support « full javascript » website, this is why the paperarchive.space doesn't work.

Sorry

[ishannaktode]

+1
Same issue
No error log

[mgifford]

This looked pretty hopeful, but didn't work for me either. This isn't a full headless site by any means.

$ python3 main.py --domain https://canada.ca --output sitemap.xml --report
Number of found URL : 1
Number of links crawled : 1
Mikes-MBP-3:python-sitemap mikegifford$ cat sitemap.xml 
<?xml version="1.0" encoding="UTF-8"?>
<urlset
      xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
      xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9
            http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">

</urlset>

But maybe this helps.

$ python3 main.py --domain https://canada.ca --output sitemap.xml --debug
INFO:root:Start the crawling process
INFO:root:Crawling #0: https://canada.ca
DEBUG:root:https://canada.ca ==> <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)>
INFO:root:Crawling has reached end of all found links

[c4software]

Dug into this one. There are two separate silent-failure bugs in __crawl() that both produce "0 URLs found" with no explanation:

1. Fetch failures without an HTTP status code (SSL errors, DNS failures, timeouts, connection resets — like mgifford's [SSL: CERTIFICATE_VERIFY_FAILED] above) were only logged at debug level, invisible unless --debug is passed, and never counted anywhere — hence "no error log".
2. A response missing both Last-Modified and Date headers raised KeyError, which silently dropped the entire page (and every link on it) from the crawl, not just the optional <lastmod>.

Fixed both in #101: non-HTTP fetch failures now log at ERROR (visible by default) and are counted in a new network_errors tally shown by --report; a missing/malformed date header no longer aborts processing the page.

Note this won't fix cases where the root cause is a fully client-side-rendered site (no JS execution) — that's #91, which the same PR adds an opt-in --render-js flag for.

---

Generated by Claude Code