is my robot file correct
-
hi, can anyone let me know if my robot file is correct. my pages and wordpress posts are being indexed but not showing in serps and wondering if my robot file is wrong
-
Yes, your robots.txt file appears to be correctly configured for common use cases. Here is a brief explanation of each line to ensure it meets your needs:
Sitemap: https://www.in2town.co.uk/sitemap_index.xml: This specifies the location of your sitemap, which helps search engines understand the structure of your site and find all of your pages. User-agent: *: This applies the following rules to all web crawlers. Disallow: /wp-admin/: This prevents web crawlers from accessing your WordPress admin area, which is standard practice for security and to prevent search engines from indexing unnecessary pages. Allow: /wp-admin/admin-ajax.php: This explicitly allows access to admin-ajax.php, which is often necessary for some plugins and features in WordPress to function correctly. Allow: /ads.txt: This explicitly allows access to the ads.txt file, which is used by the Authorized Digital Sellers standard to prevent ad fraud.
-
@blogwoman1 said in is my robot file correct:
hi, can anyone let me know if my robot file is correct. my pages and wordpress posts are being indexed but not showing in serps and wondering if my robot file is wrong
https://www.in2town.co.uk/robots.txtHi, to determine if your robots.txt file is correct, review the content of the file to ensure it's not inadvertently blocking important pages. Your robots.txt file should specify which parts of your site should not be crawled, while allowing access to essential pages and posts.
Here's a quick check of your file:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpThis configuration should not block search engines from indexing your posts and pages. If your pages are being indexed but not appearing in SERPs, it might be due to other factors like content quality, keyword competition, or lack of backlinks. Use Google Search Console to check for any crawl errors, indexing issues, or manual penalties. Additionally, ensure that your sitemap is properly submitted and up-to-date, and consider optimizing your content and meta tags for better visibility in search results.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Not Picking Up Posts
I am trying to work out why from March 4th Google is not seeing my posts. Our google impressions have dropped from 8,000 to 40. If you put in the full article name with speach marks it does not find it, and instead shows the home page in google. We have not had any warnings. We did have work done on our site but nothing else i could think of to cause this. Can anyone let me know what may have caused this. All articles are original
Technical SEO | | headlinesplus0 -
What Tools Do I USe To Find Why My Site No Longer Ranks
Hi, I made the mistake of hiring a freelancer to work on my website [in2town.co.uk](link url) but after having a good website things went from bad to worse. The freelancer was kicked off the platform due to lots of compliants from people and creating backdoors to websites and posting on them. It cost me money to have the back door to our site closed. I then found lots of websites were stealing my content through the rss feed. Two of those sites have now been shut down by their hosting company. With all these problems I found in Feb that the hundreds of keywords that I ranked for had vanished. And all the ones that were in the top ten for many years have also vanished. When I create an article which includes https://www.in2town.co.uk/skegness-news/lincolnshire-premier-inn-staff-fear-for-their-jobs/ they cannot be found in Google. Normally before all these problems, my articles were found straight away. If I put in the title name Lincolnshire Premier Inn Staff Fear For Their Jobs and then add In2town in front of it, then instead of the page coming up with the article, it instead shows the home page. Can anyone please advise what tools i should be using to find out the problems and solve them, and can anyone offer advice please on what to do to solve this.
Technical SEO | | blogwoman10 -
Only my homepage ranks for my keywords, should I delete my other pages?
I am an independent artist and all of my business inquiries come through my website (www.ChrisCarlsonArt.com). Over the last 6 months I have been trying to get pages other than my homepage to rank for my keywords, but I haven't made any progress. I worry that I am cannibalizing my keywords since my pages all have similar information. Should I just delete my other pages and focus on ranking my homepage? Also, if I delete my other pages will that have a negative impact on my rankings?
SEO Tactics | | PunchyMcSkeletor0 -
words not appearing in the search engine or not ranking correctly ?
Although we do all the SEO work on our website, we do not rank in the words computer repair and laptop repair, although our backlink rate is good. The address of our website is https://trueonefix.com, one of the keywords I gave to the homepage that I was really curious about, for example, computer repair, I created a page on this page ( https://trueonefix.com/computer-repair/ ), they told me it was cannibalization and I removed it, now the site is getting less rankings than before. Can anyone help as to why this is? No matter what I do, I can't rank the words. #trueonefix #googlerank
SEO Tactics | | Trueonefix0 -
Blocking in Robots.txt and the re-indexing - DA effects?
I have two good high level DA sites that target the US (.com) and UK (.co.uk). The .com ranks well but is dormant from a commercial aspect - the .co.uk is the commercial focus and gets great traffic. Issue is the .com ranks for brand in the UK - I want the .co.uk to rank for brand in the UK. I can't 301 the .com as it will be used again in the near future. I want to block the .com in Robots.txt with a view to un-block it again when I need it. I don't think the DA would be affected as the links stay and the sites live (just not indexed) so when I unblock it should be fine - HOWEVER - my query is things like organic CTR data that Google records and other factors won't contribute to its value. Has anyone ever blocked and un-blocked and whats the affects pls? All answers greatly received - cheers GB
Technical SEO | | Bush_JSM0 -
Unsolved Moz crawler not crawling on my site
Hi all, im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. This is my robots.txt file : https://www.wearefutureheads.com/robots.txt I'm not sure what else am I missing.. can anyone help
Product Support | | teikh0 -
Unsolved What would the exact text be for robots.txt to stop Moz crawling a subdomain?
I need Moz to stop crawling a subdomain of my site, and am just checking what the exact text should be in the file to do this. I assume it would be: User-agent: Moz
Getting Started | | Simon-Plan
Disallow: / But just checking so I can tell the agency who will apply it, to avoid paying for their time with the incorrect text! Many thanks.0