Getting Slammed by Amazon S3 Bills?

8 years ago

To answer your question: no, I've never experienced what you're describing. My S3 usage has never done anything unusual like that. And I've never heard of any issues like you describe with Chrome either.

It sounds to me like you either have a bug in your application or you have bots or other traffic accessing your data and running up your bill. With that in mind, I'm not surprised Amazon doesn't want to do anything -- it's not really their job to diagnose problems with your application or secure your site from bots. Moreover, their advice to go through your logs (as unsatisfying as it might be) isn't bad.

If I were you I'd look into something like Loggly, Sumo Logic, Logstash or one of the multitudes of others out there to try to figure this out. You should be able to quickly parse and search those logs, looking for instances where there are multiple downloads from the same IP (if that's really what's happening) or perhaps downloads from bots (if that's what's going on). You can test your Chrome theory by downloading your own file with Chrome and looking at the logs for traffic coming from your IP address. Honestly, I don't have high hopes that Chrome is causing this problem, but it shouldn't be hard to test.

As for fixing it, it all depends on you and your application. If it's truly a Chrome bug, you can detect the User Agent and do something special for them. If it's bots, you'll need to tighten up your access policies. If it's something else, you'll need to fix your application.

I realize my answer isn't probably the answer you're looking for but I really do think this is more likely to be a problem with your application than with S3 or Chrome. I'd also suggest you post this question over at a place Stack Overflow -- though to be fair, they'll need a lot more specific, technical information to be able to help you. (How are people accessing S3? What programming languages are you getting? What error messages are you getting? Log results? Etc. etc. etc.) As it is, all anyone can do is conjecture.

[ 1 ] Thanks
2 replies

Signature

<< This is not a sales pitch >>

{{ DiscussionBoard.errors[10401893].message }}

David Beroff

8 years ago

Sorry if this reply is way too simplistic, but sometimes what's obvious to one person isn't to another. Have you considered simply hosting the video on YouTube?

[ 1 ] Thanks
1 reply

Signature

Put MY voice on YOUR video: AwesomeAmericanAudio.com

{{ DiscussionBoard.errors[10401908].message }}

DavidBlaise

8 years ago

Originally Posted by David Beroff

Sorry if this reply is way too simplistic, but sometimes what's obvious to one person isn't to another. Have you considered simply hosting the video on YouTube?

Certainly a valid question and I DO prefer simple! We've considered YouTube and we've hosted some videos there. Problem is that people see the YouTube link in the lower right and often click through to see what else you have on YouTube. Once there, it's easy to get distracted by all the "related videos" and other shiny objects and quickly forget why they were on your site to begin with. That's why we would prefer an option that keeps them on our site free of distractions.

Another simple approach we've tried is hosting the videos on our own servers. However, that becomes slow and is not scalable when a lot of people hit it at the same time.

Thanks
1 reply

{{ DiscussionBoard.errors[10403467].message }}

David Beroff

8 years ago

Originally Posted by DavidBlaise

Certainly a valid question and I DO prefer simple! We've considered YouTube and we've hosted some videos there. Problem is that people see the YouTube link in the lower right and often click through to see what else you have on YouTube. Once there, it's easy to get distracted by all the "related videos" and other shiny objects and quickly forget why they were on your site to begin with. That's why we would prefer an option that keeps them on our site free of distractions.

Understood and respected. YouTube has a number of switches that you can use to turn most (not all) of the distractions off. Vimeo has a paid option that might also quickly meet your needs; not familiar with it.

Originally Posted by DavidBlaise

Another simple approach we've tried is hosting the videos on our own servers. However, that becomes slow and is not scalable when a lot of people hit it at the same time.

Well, that's not an automatic conclusion, (with all due respect); it's only when you use servers that weren't really intended for that sort of load. You need to use a beefier hosting solution... which is exactly what you are/were doing with Amazon S3. i.e., You're paying for them to deliver a large amount of data quickly, and that's exactly what they are doing.

So their suggestion to go through the logs is actually reasonable, but you don't have to do so manually; there's quite a bit of software that can do this. My (uninformed) hunch is that the others above are likely correct, that you have content which likely is supposed to be behind a paywall, but you may have uninvited guests. If that's the case, you may have to randomize the URL's, so as to give each paying customer their own, private access; yes, there's software to do this, as well.

If (for some unlikely reason) the issue really is with S3, (which I highly doubt), I can get you another similar-class host, (Rackspace), for $5/mo. PM me, since I don't feel like getting banned again for answering a relevant question with a relevant answer which includes a URL.

Thanks

Signature

Put MY voice on YOUR video: AwesomeAmericanAudio.com

{{ DiscussionBoard.errors[10403521].message }}

DavidBlaise

8 years ago

Originally Posted by kilgore

To answer your question: no, I've never experienced what you're describing. My S3 usage has never done anything unusual like that. And I've never heard of any issues like you describe with Chrome either.

It sounds to me like you either have a bug in your application or you have bots or other traffic accessing your data and running up your bill. With that in mind, I'm not surprised Amazon doesn't want to do anything -- it's not really their job to diagnose problems with your application or secure your site from bots. Moreover, their advice to go through your logs (as unsatisfying as it might be) isn't bad.

If I were you I'd look into something like Loggly, Sumo Logic, Logstash or one of the multitudes of others out there to try to figure this out. You should be able to quickly parse and search those logs, looking for instances where there are multiple downloads from the same IP (if that's really what's happening) or perhaps downloads from bots (if that's what's going on). You can test your Chrome theory by downloading your own file with Chrome and looking at the logs for traffic coming from your IP address. Honestly, I don't have high hopes that Chrome is causing this problem, but it shouldn't be hard to test.

As for fixing it, it all depends on you and your application. If it's truly a Chrome bug, you can detect the User Agent and do something special for them. If it's bots, you'll need to tighten up your access policies. If it's something else, you'll need to fix your application.

I realize my answer isn't probably the answer you're looking for but I really do think this is more likely to be a problem with your application than with S3 or Chrome. I'd also suggest you post this question over at a place Stack Overflow -- though to be fair, they'll need a lot more specific, technical information to be able to help you. (How are people accessing S3? What programming languages are you getting? What error messages are you getting? Log results? Etc. etc. etc.) As it is, all anyone can do is conjecture.

Thank you for your thorough response. You've certainly given us some good places to start. We have not yet begun to fight!

Thanks

{{ DiscussionBoard.errors[10403499].message }}

Rob Whisonant

8 years ago

It really sounds more like your S3 bucket has been listed on blackhat and file sharing sites.

Re's
Rob Whisonant

[ 1 ] Thanks
1 reply

Signature

FREE Email Marketing Quick Start Course! Start Building a Profitable Email List Today!

{{ DiscussionBoard.errors[10402091].message }}

DavidBlaise 8 years ago

Originally Posted by Rob Whisonant

It really sounds more like your S3 bucket has been listed on blackhat and file sharing sites.

That's what we thought initially. However, from what we have been told, the log files don't appear to support that theory. Still investigating, however. Thank you!
- Thanks
{{ DiscussionBoard.errors[10403488].message }}

Market4Keyword

8 years ago

You wrote that Youtube is not good enough since people click on
the "Youtube" logo and then get distracted by all the related videos
shown.

Just a suggestion for you : check out Vimeo.com . Similar to Youtube.
They also offer a "Pro" service. Even Ebay uses their service.
Really good for video hosting. And it's not too expensive as well.

https://vimeo.com/pro

You can try them for free.

Thanks

{{ DiscussionBoard.errors[10403581].message }}

kimanierick

8 years ago

if there is such a problem, then I have not noted it or am too ignorant. Will try and see if your theory might be true. I suggest then you consider vimeo. it might for you since you have discredited you tube which would have been the best resort. i hope you will find the best option. thanks