Include iframe contents

[marshvee]

Iframes are sometimes used to have parts of sites controlled by a CMS.

Here we add the option of inspecting the iframe's content and for any links that are to the site being indexed, having those included. It takes into account the tag, since if the base tag matches the site being indexed, then all relative URLs should be crawled.

If you want to enable this option, you can just add the flag:

python main.py --fetch-iframes

Resolves #90

[c4software]

Reviewed and tested locally against a small mock site (main domain + a "CMS" iframe on a different host declaring a matching <base href>, plus an unrelated third-party iframe with no <base> tag):

• --fetch-iframes off (default): behavior is unchanged — only same-domain <a> links are crawled, iframe srcs are never fetched. Backward compatible.
• --fetch-iframes on: the iframe pointing to the unrelated external domain is fetched once and correctly dropped (no matching <base>, so it's excluded from the sitemap and not crawled further). The iframe with a <base href> matching the target domain is correctly detected, and its relative links are resolved against that base and added to the crawl — exactly the behavior requested in #90.

Code looks solid: it's opt-in, respects existing --exclude/--skipext/robots.txt checks for iframe-discovered links, and doesn't change output for sites that don't use iframes. Left one minor whitespace nit inline (non-blocking).

---

Generated by Claude Code

[c4software]

Nit: this comment line mixes 2 spaces with tabs before the # (the rest of the file uses tabs only). Doesn't break anything since comment-only lines don't affect Python's indentation, but worth a quick cleanup for consistency with the rest of the file.

Suggested change

	# In this case: if the base tag matches the site being indexed, then all relative URLs should be crawled.
	# In this case: if the base tag matches the site being indexed, then all relative URLs should be crawled.

---

Generated by Claude Code