Please help me with this twitter scraper script app

7 replies
Hey everyone,

So I found this script from like 2011 or something, which uses a Google doc spreadsheet with script to scrape the followers or following of a Twitter account.

So far I have been able to get it to scrape 5,000 users, but it will not go past that, and when I run it again, it just scrapes the same 5,000 again.

I have a client who is following 40,000 people and he would like his whitelist (people he never wants to unfollow) to be made up of only verified users, and this scraper has a column that shows if the user is verified.

Does anyone know of any code I can add to the script so that when it runs the second time, it starts up where it left off?

Here are the original links about the script:

https://mashe.hawksey.info/2011/03/e...ter-followers/

https://mashe.hawksey.info/2011/08/e...#comment-12920

Here is the link with the added script to try to get the scraping to continue without stopping (I'm not even sure if this is accomplishing anything, since Twitter is just doing the 5000 limit)

https://gist.github.com/mhawksey/1121340

Also, when I use the Google doc spreadsheet, when I am on the script editing page, and I click "Run", it comes up with this error:

ScriptProperties API is deprecated.
File: Code Line: 501
The API has been marked as deprecated which means that the feature should be avoided and may be removed in the future. Consider using an alternative solution.

This is line 501: if (!ScriptProperties.getProperty("cursor")){

But when I am in the spreadsheet and I use the spreadsheet's function to scrape the "friends" of the account, it will give me the first 5,000 people each time.

I'm not a developer, but I did have to create the twitter app to make this work. I don't understand programming/scripts, etc, so I can't fix this to make it work right. I believe it worked properly in 2011 when the guy made it.

I do understand twitter's API limits, but I was just wondering if anyone knew how I could edit the script so that when I ran it again, it would start back up where I left off once I hit the limit and waited, instead of it just doing the first 5,000 each time. I was hoping I could get the first 5,000, then wait and run it again to get 5,001-10,000, and so on...

Thanks to anyone who can help!
#app #scraper #script #twitter
  • Profile picture of the author BranTheMan
    You might want to try stackoverflow to get help with this one. Some of the people who frequent this section couldn't even tell you what a Twitter card is, let alone javascript.
    Signature
    Buyral Media Web Marketing Services.
    {{ DiscussionBoard.errors[10363876].message }}
    • Profile picture of the author Lisha5684
      Originally Posted by BranTheMan View Post

      You might want to try stackoverflow to get help with this one. Some of the people who frequent this section couldn't even tell you what a Twitter card is, let alone javascript.
      Haha! Thanks Brantheman. I will try stackoverflow.
      Signature

      -Start Getting Noticed On Twitter: http://twonderwoman.com
      -Visit my blog: http://blogandretire.com/blog - Tips for blogging, social media marketing, and online business.

      {{ DiscussionBoard.errors[10364076].message }}
  • Profile picture of the author Joe Ray
    Did you get any answers on stackoverflow? Is your problem solved?
    {{ DiscussionBoard.errors[10365740].message }}
    • Profile picture of the author Lisha5684
      Originally Posted by Joe Ray View Post

      Did you get any answers on stackoverflow? Is your problem solved?
      Hi Joe, not yet. I haven't gotten a chance to go over there and ask. I'm not registered there, so I haven't gotten around to making an account and posting my question. Thanks for asking

      At this moment, this is not an urgent matter. It was the other day, but my client said don't worry about it. I still do want to figure this out at some point though because I have had more than one client ask in the past that I not unfollow verified accounts they're following.
      Signature

      -Start Getting Noticed On Twitter: http://twonderwoman.com
      -Visit my blog: http://blogandretire.com/blog - Tips for blogging, social media marketing, and online business.

      {{ DiscussionBoard.errors[10373948].message }}
    • Profile picture of the author Lisha5684
      Originally Posted by Joe Ray View Post

      Did you get any answers on stackoverflow? Is your problem solved?
      I decided to go over and do it now, and it turns out I did have an account with stackoverflow. I must have asked a question on there a long time ago... Anyway, I posted my question, so hopefully I'll get an answer.
      Signature

      -Start Getting Noticed On Twitter: http://twonderwoman.com
      -Visit my blog: http://blogandretire.com/blog - Tips for blogging, social media marketing, and online business.

      {{ DiscussionBoard.errors[10373971].message }}
      • Profile picture of the author Joe Ray
        Originally Posted by Lisha5684 View Post

        I decided to go over and do it now, and it turns out I did have an account with stackoverflow. I must have asked a question on there a long time ago... Anyway, I posted my question, so hopefully I'll get an answer.
        I looked at the code you used, I wonder why you can't just ask the same coder who wrote that code to modify your existing code to solve your problem?
        {{ DiscussionBoard.errors[10374276].message }}
        • Profile picture of the author Lisha5684
          Originally Posted by Joe Ray View Post

          I looked at the code you used, I wonder why you can't just ask the same coder who wrote that code to modify your existing code to solve your problem?
          Genius. Sometimes I lose my head when I've spent all day at the computer, haha! Forums and emailing bloggers are usually a last resort to me.
          Signature

          -Start Getting Noticed On Twitter: http://twonderwoman.com
          -Visit my blog: http://blogandretire.com/blog - Tips for blogging, social media marketing, and online business.

          {{ DiscussionBoard.errors[10374324].message }}

Trending Topics