10 {{ upvoteCount | shortNum }}
10 {{ upvoteCount | shortNum }}

Wordpress: Disable Bot from Crawling? (setting up new site)

by PrincessJasmine
8 replies
  • SEO
    • |
Whats the best practice when setting up a NEW wordpress website.

Is it better to set it up offline, have it fully designed and populated with contents OFFLINE then move it ONLINE

OR

Set up wordpress online. Go to setting and disable bots from crawling. Put in the contents, then when ready, enable crawling again?


Thanks
#bot #crawling #disable #wordpress
  • Profile picture of the author Barry Unruh
    Whether it is best or not... I go with option 2
    Signature
    Brain Drained...Signature Coming Soon!
    {{ DiscussionBoard.errors[5209244].message }}
  • Profile picture of the author GeorgR.
    I never "set up" a blog offline.

    Each time you make a new post it pings and lures in spiders telling them you have new content. And this is a good thing!

    Actually..thinking about it...the idea of setting a blog up "offline" with several pieces of content already filled is REALLY BAD.
    Signature
    *** Affiliate Site Quick --> The Fastest & Easiest Way to Make Affiliate Sites!<--
    -> VISIT www.1UP-SEO.com *** <- Internet Marketing, SEO Tips, Reviews & More!! ***
    *** HIGH QUALITY CONTENT CREATION +++ Manual Article Spinning (Thread Here) ***
    Content Creation, Blogging, Articles, Converting Sales Copy, Reviews, Ebooks, Rewrites
    {{ DiscussionBoard.errors[5209331].message }}
  • Profile picture of the author PrincessJasmine
    Ah I see.
    Thanks guys!

    Should I even 'disable' the crawling at all then? (even tho my site isnt fully designed and with zero contents at the moment?)
    {{ DiscussionBoard.errors[5211081].message }}
  • Profile picture of the author twotoofs
    I have started stopping the crawling of my news sites until I have them set up with one piece of content. I'm not far enough along to see if it has any effect of things though.
    {{ DiscussionBoard.errors[5212057].message }}
  • Profile picture of the author Power Solutions
    Personally I do not see the reasoning behind Google to crawl your website when you have not optimized for SEO / content / whatever. So I always put up a coming soon plugin while I set things up properly. I put up 2-3 SEO optimized posts, and have the appropriate info pages available that are required for adsense before I take down the "Coming Soon" plugin.

    If there is any particular reason why it's beneficial for crawlers to visit an empty website please do let me know.
    Signature
    -=: Internet Marketing News :=------=: Resources I Use :=------=: Prime Ad Network :=------=: ?Profit? :=-
    {{ DiscussionBoard.errors[5212567].message }}
    • Profile picture of the author PrincessJasmine
      Originally Posted by khawaja1971 View Post

      Personally I do not see the reasoning behind Google to crawl your website when you have not optimized for SEO / content / whatever. So I always put up a coming soon plugin while I set things up properly. I put up 2-3 SEO optimized posts, and have the appropriate info pages available that are required for adsense before I take down the "Coming Soon" plugin.

      If there is any particular reason why it's beneficial for crawlers to visit an empty website please do let me know.
      Thanks everyone for the input.
      I used to put 'coming soon' and fancy countdown plugins too.
      However, I soon realize that if Google happens to crawl my page and read the 'coming soon' sign, then put that into their cache... I am afraid it will have cause some negative impact.

      In another words, my site will be 'coming soon' forever until they decide to crawl my site again?

      Thanks
      {{ DiscussionBoard.errors[5214021].message }}
  • Profile picture of the author GeorgR.
    I am also "paranoid" if i work on a site or page, i usually disable crawlers, either within wordpress or using a robots.txt file to exclude all spiders. And then only open it up when i know that the page/article is "perfect". (But only on a brand-new site when i make the first post like the home page)

    But then i am pretty sure that Google is aware that sites usually do not have "perfect" content right away the second they go online - people are editing and working on sites. This is the reason there is always a delay until Google finally comes and indexes, after some days, and not right away.

    I think its overrated tho since Google will re-visit...it's just something you have to give a few days...this is just how it works.

    (When i make a new post...which could be 400-500 edits on a *complicated* page with PHP/forms etc...it always save as "Draft"...and once i think it's about in the final state i publish, when i know that the main things are right like tags, headlines etc.)
    Signature
    *** Affiliate Site Quick --> The Fastest & Easiest Way to Make Affiliate Sites!<--
    -> VISIT www.1UP-SEO.com *** <- Internet Marketing, SEO Tips, Reviews & More!! ***
    *** HIGH QUALITY CONTENT CREATION +++ Manual Article Spinning (Thread Here) ***
    Content Creation, Blogging, Articles, Converting Sales Copy, Reviews, Ebooks, Rewrites
    {{ DiscussionBoard.errors[5214263].message }}
    • Profile picture of the author BigNorm
      Originally Posted by GeorgR. View Post

      I am also "paranoid" if i work on a site or page, i usually disable crawlers, either within wordpress or using a robots.txt file to exclude all spiders. And then only open it up when i know that the page/article is "perfect". (But only on a brand-new site when i make the first post like the home page)

      But then i am pretty sure that Google is aware that sites usually do not have "perfect" content right away the second they go online - people are editing and working on sites. This is the reason there is always a delay until Google finally comes and indexes, after some days, and not right away.

      I think its overrated tho since Google will re-visit...it's just something you have to give a few days...this is just how it works.

      (When i make a new post...which could be 400-500 edits on a *complicated* page with PHP/forms etc...it always save as "Draft"...and once i think it's about in the final state i publish, when i know that the main things are right like tags, headlines etc.)

      Excellent advice which I work by as well. When I set up a new blog I disable the bots from crawling my site within Wordpress till I have at least 2 pages of content 500+ words, Privacy Policy, About and Contact Page and an additional 3 pages of content 500+ scheduled for posting every 3-4 days set up. This gives me time to write additional content while my site is updating constantly.
      {{ DiscussionBoard.errors[5214482].message }}

Trending Topics