dlvr.it Feed Fetcher is how dlvr.it grabs RSS or Atom feeds when users add them to their dlvr.it. dlvr.it Feed Fetcher collects and periodically refreshes these user-initiated feeds. Find answers below to some of the most commonly asked questions about how this user-controlled feed retriever works.
How do I request that dlvr.it Feed Fetcher not retrieve some or all of my site's feeds?
To prevent dlvr.it Feed Fetcher from retrieving your feeds update your robots.txt file.
To prevent accidental blocking, dlvr.it does not pay attention to the "User-Agent: *" section of robots.txt. A dlvr.it user explicitly tells dlvr.it to retrieve a feed URL and we don't want to make it difficult on your users to access your RSS feeds with dlvr.it Feed Fetcher if you have blocked certain sections for general search-engine purposes.
To block dlvr.it Feed Fetcher enter:
User-Agent: dlvr.it
Disallow: /path/to/your/feeds
As with any other agent, adding "Disallow: /" will prevent dlvr.it Feed Fetcher from retrieving any content on your site.
The "dlvr.it" agent will also pay attention to "Allow: directives. This means that you can disallow some parts of your site while allowing others to be read.
User-Agent: dlvr.it
Disallow: /
Allow: /path/to/some/feeds
A plain "Disallow:" with no path means "Allow all"
User-Agent: dlvr.it
Disallow:
How often will dlvr.it Feed Fetcher retrieve my feeds?
dlvr.it Feed Fetcher shouldn't retrieve feeds from most sites more than once every hour on average. Some frequently updated sites may be refreshed more often. Note, however, that due to network delays, it's possible that Fetcher may briefly appear to retrieve your feeds more frequently.
Why is dlvr.it Feed Fetcher trying to download incorrect links from my server, or from a server that doesn't exist?
dlvr.it Fetcher retrieves feeds at the request of users who have added them to their dlvr.it. It is possible that a user has requested a feed URL location that does not exist.
Why is dlvr.it Feed Fetcher downloading information from our "secret" web server?
dlvr.it Feed Fetcher retrieves feeds at the request of users who have added them to their dlvr.it. It is possible that the request came from a user who knows about your "secret" server or typed it in by mistake.
Why are there hits from multiple machines at dlvr.it, all with user-agent dlvr.it/1.0?
dlvr.it Feed Fetcher was designed to be distributed on several machines to improve performance and scale as the web grows.
Can you tell me the IP addresses from which Fetcher makes requests so that I can filter my logs?
The IP addresses used by dlvr.it Feed Fetcher change from time to time. The best way to identify accesses by dlvr.it Feed Fetcher is to use its identifiable user-agent: dlvr.it/1.0.
Why is dlvr.it Feed Fetcher downloading the same page on my site multiple times?
In general, dlvr.it Feed Fetcher should only download one copy of each file from your site during a given feed retrieval. Sometimes, the same feed has multiple ids. In that case, it might be extracted multiple times. We are working on improving our deduplication technology to minimize the number of update requests dlvr.it Feed Fetcher performs.
Do you support push technology?
Yes, dlvr.it supports push hubs. If your feeds advertise a push hub, dlvr.it will subscribe for updates and reduce the number of polls to once a day.
My dlvr.it Feed Fetcher question isn't answered here. Where can I get more help?
If you're still having trouble, you can reach out to us at support @ dlvrit.com